==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-7dccb59994-hvp6g Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Thu, 18 Apr 2024 23:03:51 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=7dccb59994 skaffold.dev/run-id=0974b4ac-b0ac-4db2-a16e-bc89d496a673 Annotations: Status: Running IP: 10.106.42.184 IPs: IP: 10.106.42.184 Controlled By: ReplicaSet/lodemon-7dccb59994 Containers: lodemon: Container ID: containerd://fe19947ec4740ce27d675ac0eaa3568b85aca59908bcbe0996249796a9db8bc5 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:1a2eb1b82876135a9f9dd8f7140893253f927bd2dd85e927623ff2af101538dd Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Thu, 18 Apr 2024 23:04:08 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-prrtg (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-prrtg: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 14m default-scheduler Successfully assigned xlou/lodemon-7dccb59994-hvp6g to gke-xlou-cdm-default-pool-1b0769c8-7vmj Normal Pulling 14m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 14m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 16.591817872s (16.591825879s including waiting) Normal Created 14m kubelet Created container lodemon Normal Started 14m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.42.184:8080 Press CTRL+C to quit 00:05:08 INFO 00:05:08 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:08 INFO [loop_until]: OK (rc = 0) 00:05:08 DEBUG --- stdout --- 00:05:08 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:08 DEBUG --- stderr --- 00:05:08 DEBUG 00:05:09 INFO 00:05:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:09 INFO [loop_until]: OK (rc = 0) 00:05:09 DEBUG --- stdout --- 00:05:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:09 DEBUG --- stderr --- 00:05:09 DEBUG 00:05:09 INFO 00:05:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:09 INFO [loop_until]: OK (rc = 0) 00:05:09 DEBUG --- stdout --- 00:05:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:09 DEBUG --- stderr --- 00:05:09 DEBUG 00:05:09 INFO 00:05:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:09 INFO [loop_until]: OK (rc = 0) 00:05:09 DEBUG --- stdout --- 00:05:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:09 DEBUG --- stderr --- 00:05:09 DEBUG 00:05:09 INFO 00:05:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:09 INFO [loop_until]: OK (rc = 0) 00:05:09 DEBUG --- stdout --- 00:05:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:09 DEBUG --- stderr --- 00:05:09 DEBUG 00:05:09 INFO 00:05:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:09 INFO [loop_until]: OK (rc = 0) 00:05:09 DEBUG --- stdout --- 00:05:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:09 DEBUG --- stderr --- 00:05:09 DEBUG 00:05:09 INFO 00:05:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:09 INFO [loop_until]: OK (rc = 0) 00:05:09 DEBUG --- stdout --- 00:05:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:09 DEBUG --- stderr --- 00:05:09 DEBUG 00:05:09 INFO 00:05:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:09 INFO [loop_until]: OK (rc = 0) 00:05:09 DEBUG --- stdout --- 00:05:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:09 DEBUG --- stderr --- 00:05:09 DEBUG 00:05:09 INFO 00:05:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:10 INFO [loop_until]: OK (rc = 0) 00:05:10 DEBUG --- stdout --- 00:05:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:10 DEBUG --- stderr --- 00:05:10 DEBUG 00:05:10 INFO 00:05:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:10 INFO [loop_until]: OK (rc = 0) 00:05:10 DEBUG --- stdout --- 00:05:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:10 DEBUG --- stderr --- 00:05:10 DEBUG 00:05:10 INFO 00:05:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:10 INFO [loop_until]: OK (rc = 0) 00:05:10 DEBUG --- stdout --- 00:05:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:10 DEBUG --- stderr --- 00:05:10 DEBUG 00:05:10 INFO 00:05:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:10 INFO [loop_until]: OK (rc = 0) 00:05:10 DEBUG --- stdout --- 00:05:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:10 DEBUG --- stderr --- 00:05:10 DEBUG 00:05:10 INFO 00:05:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:10 INFO [loop_until]: OK (rc = 0) 00:05:10 DEBUG --- stdout --- 00:05:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:10 DEBUG --- stderr --- 00:05:10 DEBUG 00:05:10 INFO 00:05:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:10 INFO [loop_until]: OK (rc = 0) 00:05:10 DEBUG --- stdout --- 00:05:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:10 DEBUG --- stderr --- 00:05:10 DEBUG 00:05:10 INFO 00:05:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:10 INFO [loop_until]: OK (rc = 0) 00:05:10 DEBUG --- stdout --- 00:05:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:10 DEBUG --- stderr --- 00:05:10 DEBUG 00:05:10 INFO 00:05:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:10 INFO [loop_until]: OK (rc = 0) 00:05:10 DEBUG --- stdout --- 00:05:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:10 DEBUG --- stderr --- 00:05:10 DEBUG 00:05:11 INFO 00:05:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:11 INFO [loop_until]: OK (rc = 0) 00:05:11 DEBUG --- stdout --- 00:05:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:11 DEBUG --- stderr --- 00:05:11 DEBUG 00:05:11 INFO 00:05:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:11 INFO [loop_until]: OK (rc = 0) 00:05:11 DEBUG --- stdout --- 00:05:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:11 DEBUG --- stderr --- 00:05:11 DEBUG 00:05:11 INFO 00:05:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:11 INFO [loop_until]: OK (rc = 0) 00:05:11 DEBUG --- stdout --- 00:05:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:11 DEBUG --- stderr --- 00:05:11 DEBUG 00:05:11 INFO 00:05:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:11 INFO [loop_until]: OK (rc = 0) 00:05:11 DEBUG --- stdout --- 00:05:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:11 DEBUG --- stderr --- 00:05:11 DEBUG 00:05:11 INFO 00:05:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:11 INFO [loop_until]: OK (rc = 0) 00:05:11 DEBUG --- stdout --- 00:05:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:11 DEBUG --- stderr --- 00:05:11 DEBUG 00:05:11 INFO 00:05:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:11 INFO [loop_until]: OK (rc = 0) 00:05:11 DEBUG --- stdout --- 00:05:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:11 DEBUG --- stderr --- 00:05:11 DEBUG 00:05:11 INFO 00:05:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:11 INFO [loop_until]: OK (rc = 0) 00:05:11 DEBUG --- stdout --- 00:05:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:11 DEBUG --- stderr --- 00:05:11 DEBUG 00:05:12 INFO 00:05:12 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:12 INFO [loop_until]: OK (rc = 0) 00:05:12 DEBUG --- stdout --- 00:05:12 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:12 DEBUG --- stderr --- 00:05:12 DEBUG 00:05:12 INFO 00:05:12 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:12 INFO [loop_until]: OK (rc = 0) 00:05:12 DEBUG --- stdout --- 00:05:12 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:12 DEBUG --- stderr --- 00:05:12 DEBUG 00:05:12 INFO 00:05:12 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:12 INFO [loop_until]: OK (rc = 0) 00:05:12 DEBUG --- stdout --- 00:05:12 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:12 DEBUG --- stderr --- 00:05:12 DEBUG 00:05:12 INFO 00:05:12 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:05:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:12 INFO [loop_until]: OK (rc = 0) 00:05:12 DEBUG --- stdout --- 00:05:12 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:05:12 DEBUG --- stderr --- 00:05:12 DEBUG 00:05:12 INFO Initializing monitoring instance threads 00:05:12 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 00:05:12 INFO Starting instance threads 00:05:12 INFO 00:05:12 INFO Thread started 00:05:12 INFO [loop_until]: kubectl --namespace=xlou top node 00:05:12 INFO 00:05:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:12 INFO Thread started 00:05:12 INFO [loop_until]: kubectl --namespace=xlou top pods 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG --- status code --- 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG --- status code --- 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 DEBUG --- status code --- 00:05:12 INFO Thread started 00:05:12 DEBUG --- status code --- 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG --- status code --- 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 INFO Thread started 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG --- http response --- 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG --- http response --- 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG { 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG { 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 INFO Thread started 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:05:12 DEBUG { 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG { 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG { 00:05:12 DEBUG { 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG --- status code --- 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 DEBUG { 00:05:12 INFO Thread started 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "data": { 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:05:12 DEBUG "data": { 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:05:12 DEBUG { 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 INFO [loop_until]: OK (rc = 0) 00:05:12 DEBUG "status": "success", 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 DEBUG --- status code --- 00:05:12 INFO Thread started 00:05:12 DEBUG { 00:05:12 DEBUG { 00:05:12 DEBUG "data": { 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG "result": [ 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG { 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:05:12 DEBUG "data": { 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:05:12 DEBUG "data": { 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG --- stdout --- 00:05:12 DEBUG "data": { 00:05:12 DEBUG "resultType": "vector", 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG "result": [ 00:05:12 DEBUG { 00:05:12 DEBUG { 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG "data": { 00:05:12 INFO [loop_until]: OK (rc = 0) 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG { 00:05:12 DEBUG "data": { 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:05:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 16m 3233Mi am-776d56b7dd-m7qb7 16m 2535Mi am-776d56b7dd-xr8wp 15m 2945Mi ds-cts-0 6m 7726Mi ds-cts-1 4m 7500Mi ds-cts-2 8m 7723Mi ds-idrepo-0 10m 3249Mi ds-idrepo-1 8m 2884Mi ds-idrepo-2 8m 2985Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1169Mi idm-b85f9d978-wnhxb 4m 1214Mi lodemon-7dccb59994-hvp6g 2m 55Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1m 1472Mi 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG "result": [ 00:05:12 DEBUG "data": { 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 DEBUG "data": { 00:05:12 INFO Thread started 00:05:12 DEBUG "result": [ 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG { 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "metric": { 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG --- stdout --- 00:05:12 DEBUG "data": { 00:05:12 DEBUG { 00:05:12 DEBUG "result": [ 00:05:12 DEBUG { 00:05:12 DEBUG "result": [ 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG --- stderr --- 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG { 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG { 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713481512" 00:05:12 DEBUG --- http response --- 00:05:12 INFO Thread started 00:05:12 DEBUG "metric": { 00:05:12 DEBUG "data": { 00:05:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 86m 0% 5977Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 6619Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 49m 0% 2584Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 98m 0% 6514Mi 5% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 10139Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 54m 0% 2587Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 6012Mi 5% gke-xlou-cdm-ds-0f601751-hmm7 67m 0% 4661Mi 4% gke-xlou-cdm-ds-0f601751-j488 60m 0% 6008Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 11271Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 91m 0% 6482Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 59m 0% 9574Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 67m 0% 5251Mi 8% 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG { 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG { 00:05:12 DEBUG "data": { 00:05:12 DEBUG "result": [ 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG { 00:05:12 DEBUG 00:05:12 DEBUG "metric": { 00:05:12 DEBUG "result": [ 00:05:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:05:12 DEBUG "result": [ 00:05:12 DEBUG "metric": { 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481512" 00:05:12 INFO Thread started 00:05:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG }, 00:05:12 DEBUG "result": [ 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG "result": [ 00:05:12 DEBUG --- stderr --- 00:05:12 DEBUG "data": { 00:05:12 DEBUG "metric": { 00:05:12 DEBUG "data": { 00:05:12 DEBUG "metric": { 00:05:12 DEBUG "resultType": "vector", 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG { 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:05:12 DEBUG { 00:05:12 DEBUG { 00:05:12 DEBUG "instance": "10.142.0.107:9100" 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG }, 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481512" 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG "result": [ 00:05:12 INFO Thread started 00:05:12 DEBUG "value": [ 00:05:12 DEBUG { 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG { 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:05:12 DEBUG 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:05:12 DEBUG "result": [ 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG "metric": { 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG "data": { 00:05:12 DEBUG }, 00:05:12 DEBUG "metric": { 00:05:12 DEBUG "metric": { 00:05:12 DEBUG }, 00:05:12 DEBUG { 00:05:12 DEBUG "value": [ 00:05:12 DEBUG --- status code --- 00:05:12 ERROR [http_cmd]: ERROR 00:05:12 DEBUG { 00:05:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481512" 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "metric": { 00:05:12 DEBUG { 00:05:12 DEBUG "metric": { 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG { 00:05:12 DEBUG }, 00:05:12 DEBUG "result": [ 00:05:12 DEBUG }, 00:05:12 DEBUG { 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG "instance": "10.142.0.107:9100" 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "instance": "10.142.0.107:9100" 00:05:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 ERROR --- status code --- 00:05:12 DEBUG "metric": { 00:05:12 INFO Thread started 00:05:12 DEBUG 00:05:12 DEBUG [...] 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "data": { 00:05:12 DEBUG "result": [ 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "value": [ 00:05:12 DEBUG { 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "metric": { 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG }, 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:05:12 DEBUG "result": [ 00:05:12 DEBUG { 00:05:12 DEBUG }, 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG "data": { 00:05:12 DEBUG 00:05:12 DEBUG --- http response --- 00:05:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:05:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:05:12 INFO All threads has been started 127.0.0.1 - - [19/Apr/2024 00:05:12] "GET /monitoring/start HTTP/1.1" 200 - 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:05:12 DEBUG 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG "pod": "ds-cts-0" 00:05:12 DEBUG { 00:05:12 DEBUG "data": { 00:05:12 DEBUG 00:05:12 DEBUG "metric": { 00:05:12 DEBUG 00:05:12 DEBUG "pod": "ds-idrepo-0" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG 00:05:12 DEBUG "metric": { 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG 00:05:12 ERROR --- http response --- 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:05:12 DEBUG }, 00:05:12 DEBUG --- status code --- 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG "result": [ 00:05:12 DEBUG }, 00:05:12 DEBUG "metric": { 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG 00:05:12 DEBUG "pod": "ds-idrepo-0" 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG { 00:05:12 DEBUG 00:05:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "result": [ 00:05:12 DEBUG [...] 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG "value": [ 00:05:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG { 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "pod": "ds-idrepo-0" 00:05:12 DEBUG "result": [ 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG { 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG [...] 00:05:12 DEBUG { 00:05:12 DEBUG 00:05:12 DEBUG { 00:05:12 DEBUG 00:05:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG { 00:05:12 DEBUG "metric": { 00:05:12 DEBUG [...] 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "data": { 00:05:12 DEBUG [...] 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "metric": { 00:05:12 DEBUG 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:05:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:05:12 DEBUG "metric": { 00:05:12 DEBUG [...] 00:05:12 DEBUG 00:05:12 INFO [http_cmd]: http status code OK 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "pod": "ds-cts-0" 00:05:12 DEBUG [...] 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "data": { 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG "result": [ 00:05:12 DEBUG 00:05:12 DEBUG [...] 00:05:12 DEBUG [...] 00:05:12 DEBUG 00:05:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:05:12 DEBUG 00:05:12 DEBUG "data": { 00:05:12 DEBUG 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:05:12 DEBUG }, 00:05:12 DEBUG "pod": "ds-cts-0" 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG --- status code --- 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG [...] 00:05:12 DEBUG [...] 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG "result": [ 00:05:12 DEBUG { 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG [...] 00:05:12 DEBUG { 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG "data": { 00:05:12 DEBUG }, 00:05:12 DEBUG [...] 00:05:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG { 00:05:12 DEBUG "metric": { 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "instance": "10.142.0.43:9100" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG }, 00:05:12 DEBUG "result": [ 00:05:12 DEBUG 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG --- http response --- 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG [...] 00:05:12 DEBUG 00:05:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "metric": { 00:05:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:05:12 DEBUG [...] 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG "value": [ 00:05:12 DEBUG { 00:05:12 DEBUG 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG 00:05:12 DEBUG "result": [ 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG [...] 00:05:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:05:12 DEBUG 00:05:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "pod": "ds-idrepo-2" 00:05:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG "instance": "10.142.0.43:9100" 00:05:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG "metric": { 00:05:12 DEBUG 00:05:12 DEBUG "0.027446478077586996" 00:05:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:05:12 DEBUG { 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG ----- output ----- 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "instance": "10.142.0.43:9100" 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG }, 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG }, 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG 00:05:12 DEBUG "0" 00:05:12 DEBUG "pod": "ds-idrepo-0" 00:05:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:05:12 DEBUG ] 00:05:12 DEBUG }, 00:05:12 DEBUG "metric": { 00:05:12 DEBUG 00:05:12 DEBUG "pod": "ds-cts-2" 00:05:12 DEBUG 00:05:12 DEBUG { 00:05:12 DEBUG 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "pod": "ds-idrepo-2" 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG }, 00:05:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "13716.033333333333" 00:05:12 DEBUG [...] 00:05:12 DEBUG ] 00:05:12 DEBUG }, 00:05:12 DEBUG }, 00:05:12 DEBUG } 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "instance": "10.142.0.107:9100" 00:05:12 DEBUG [...] 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG "status": "success", 00:05:12 DEBUG [...] 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG }, 00:05:12 DEBUG "0.0074357574231669514" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG }, 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG ] 00:05:12 DEBUG 00:05:12 DEBUG } 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "value": [ 00:05:12 DEBUG ] 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "pod": "ds-idrepo-2" 00:05:12 DEBUG "data": { 00:05:12 DEBUG 00:05:12 DEBUG "4783.059999738552" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG ] 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "0" 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG "0.00066666666666683" 00:05:12 DEBUG "0" 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG ] 00:05:12 DEBUG 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG "0" 00:05:12 DEBUG } 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG }, 00:05:12 DEBUG "resultType": "vector", 00:05:12 DEBUG 00:05:12 DEBUG ] 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG } 00:05:12 DEBUG "12287.066666666666" 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG ] 00:05:12 DEBUG 00:05:12 DEBUG [...] 00:05:12 DEBUG "value": [ 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "0" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG "result": [ 00:05:12 DEBUG 00:05:12 DEBUG } 00:05:12 DEBUG "0" 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG "1640685568" 00:05:12 DEBUG } 00:05:12 DEBUG [...] 00:05:12 DEBUG 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:05:12 DEBUG } 00:05:12 DEBUG "415.9182608468306" 00:05:12 DEBUG [...] 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG "pod": "ds-cts-2" 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG { 00:05:12 DEBUG "pod": "ds-cts-2" 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG "0" 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG }, 00:05:12 DEBUG ] 00:05:12 DEBUG 00:05:12 DEBUG ] 00:05:12 DEBUG 00:05:12 DEBUG }, 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG ] 00:05:12 DEBUG "metric": { 00:05:12 DEBUG }, 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG "value": [ 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG } 00:05:12 DEBUG [...] 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG } 00:05:12 DEBUG "instance": "10.142.0.107:9100" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG "0" 00:05:12 DEBUG 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG "0" 00:05:12 DEBUG "0" 00:05:12 DEBUG } 00:05:12 DEBUG }, 00:05:12 DEBUG "value": [ 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:05:12 DEBUG }, 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG "pod": "ds-cts-2" 00:05:12 DEBUG 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG 00:05:12 DEBUG } 00:05:12 DEBUG }, 00:05:12 DEBUG "value": [ 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG }, 00:05:12 DEBUG 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG "0" 00:05:12 DEBUG 00:05:12 DEBUG "value": [ 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG "instance": "10.142.0.43:9100" 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG "1" 00:05:12 DEBUG "value": [ 00:05:12 DEBUG } 00:05:12 DEBUG }, 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG [...] 00:05:12 DEBUG "0" 00:05:12 DEBUG ] 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG "value": [ 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG "0" 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG ] 00:05:12 DEBUG "257383027712" 00:05:12 DEBUG } 00:05:12 DEBUG 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG "instance": "10.142.0.43:9100" 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG }, 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG "value": [ 00:05:12 DEBUG } 00:05:12 DEBUG } 00:05:12 DEBUG 1713481512, 00:05:12 DEBUG } 00:05:12 DEBUG "0.0009333333333321056" 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG ] 00:05:12 DEBUG } 00:05:12 DEBUG } 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713481572" 00:06:12 INFO 00:06:12 INFO [loop_until]: kubectl --namespace=xlou top pods 00:06:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:06:12 ERROR [http_cmd]: ERROR 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 ERROR --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG --- status code --- 00:06:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 ERROR --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713481572" 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:06:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:06:12 INFO 00:06:12 DEBUG --- http response --- 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:06:12 DEBUG ----- output ----- 00:06:12 INFO [loop_until]: kubectl --namespace=xlou top node 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG { 00:06:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG { 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG { 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "data": { 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:06:12 DEBUG "data": { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "data": { 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG "result": [ 00:06:12 DEBUG { 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "data": { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG { 00:06:12 DEBUG "data": { 00:06:12 DEBUG "data": { 00:06:12 DEBUG { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "pod": "ds-idrepo-0" 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "result": [ 00:06:12 DEBUG }, 00:06:12 DEBUG "instance": "10.142.0.107:9100" 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "pod": "ds-idrepo-0" 00:06:12 DEBUG { 00:06:12 DEBUG "value": [ 00:06:12 DEBUG }, 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG }, 00:06:12 DEBUG "metric": { 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:06:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG "pod": "ds-idrepo-2" 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG "instance": "10.142.0.43:9100" 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "pod": "ds-idrepo-2" 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:06:12 DEBUG "0" 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:06:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG }, 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 DEBUG ] 00:06:12 DEBUG "0.02000000000000028" 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG "0" 00:06:12 DEBUG "value": [ 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG "value": [ 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG "value": [ 00:06:12 DEBUG ] 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG } 00:06:12 DEBUG "463.49754444444443" 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG "780.2666666666667" 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG "2005204992" 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG { 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "data": { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "result": [ 00:06:12 DEBUG { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:06:12 DEBUG }, 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:06:12 DEBUG }, 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG "0.7781195187091299" 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481572" 00:06:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713481572" 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG --- status code --- 00:06:12 INFO [loop_until]: OK (rc = 0) 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG --- stdout --- 00:06:12 DEBUG --- status code --- 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG --- status code --- 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG --- http response --- 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 2811m 3668Mi am-776d56b7dd-m7qb7 927m 3309Mi am-776d56b7dd-xr8wp 1784m 3351Mi ds-cts-0 65m 7729Mi ds-cts-1 126m 7502Mi ds-cts-2 304m 7728Mi ds-idrepo-0 318m 3249Mi ds-idrepo-1 134m 2884Mi ds-idrepo-2 714m 2989Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1169Mi idm-b85f9d978-wnhxb 4m 1214Mi lodemon-7dccb59994-hvp6g 22m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 3067m 1879Mi 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- stderr --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:06:12 INFO [http_cmd]: http status code OK 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG --- status code --- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG --- http response --- 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG { 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:06:12 DEBUG { 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG { 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "data": { 00:06:12 DEBUG "data": { 00:06:12 DEBUG "data": { 00:06:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:06:12 DEBUG "data": { 00:06:12 DEBUG "data": { 00:06:12 DEBUG "data": { 00:06:12 DEBUG { 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG "data": { 00:06:12 DEBUG "data": { 00:06:12 DEBUG "status": "success", 00:06:12 INFO [loop_until]: OK (rc = 0) 00:06:12 DEBUG "data": { 00:06:12 DEBUG "data": { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG ----- output ----- 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "data": { 00:06:12 DEBUG { 00:06:12 DEBUG "data": { 00:06:12 DEBUG { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "data": { 00:06:12 DEBUG --- stdout --- 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "result": [ 00:06:12 DEBUG { 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "data": { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "result": [ 00:06:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6001Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 624m 2% 6645Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 50m 0% 2597Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 2351m 7% 6700Mi 6% gke-xlou-cdm-default-pool-1b0769c8-w15q 527m 1% 10108Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 135m 0% 5167Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 53m 0% 2585Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 8312m 27% 6644Mi 5% gke-xlou-cdm-ds-0f601751-hmm7 2593m 8% 5576Mi 4% gke-xlou-cdm-ds-0f601751-j488 384m 1% 6021Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 994m 3% 11137Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 795m 2% 6512Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 691m 2% 9545Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 3258m 20% 5660Mi 9% 00:06:12 DEBUG "result": [ 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG "status": "success", 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "data": { 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "data": { 00:06:12 DEBUG { 00:06:12 DEBUG { 00:06:12 DEBUG "result": [ 00:06:12 DEBUG { 00:06:12 DEBUG --- stderr --- 00:06:12 DEBUG { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "data": { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "result": [ 00:06:12 DEBUG { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG { 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:06:12 DEBUG "instance": "10.142.0.107:9100" 00:06:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:06:12 DEBUG "resultType": "vector", 00:06:12 DEBUG "pod": "ds-cts-0" 00:06:12 DEBUG "pod": "ds-idrepo-0" 00:06:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:06:12 DEBUG { 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "result": [ 00:06:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:06:12 DEBUG "instance": "10.142.0.107:9100" 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:06:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG "result": [ 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "pod": "ds-cts-0" 00:06:12 DEBUG { 00:06:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:06:12 DEBUG { 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG { 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "pod": "ds-cts-0" 00:06:12 DEBUG }, 00:06:12 DEBUG "metric": { 00:06:12 DEBUG }, 00:06:12 DEBUG "metric": { 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG }, 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "metric": { 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "instance": "10.142.0.107:9100" 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "instance": "10.142.0.107:9100" 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "pod": "ds-idrepo-0" 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG [...] 00:06:12 DEBUG [...] 00:06:12 DEBUG "value": [ 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG [...] 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "pod": "ds-idrepo-2" 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG [...] 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:06:12 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:06:12 DEBUG "instance": "10.142.0.43:9100" 00:06:12 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:06:12 DEBUG [...] 00:06:12 DEBUG "pod": "ds-cts-2" 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:06:12 DEBUG 00:06:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG 00:06:12 DEBUG "pod": "ds-cts-2" 00:06:12 DEBUG 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:06:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 00:06:12 DEBUG "value": [ 00:06:12 DEBUG }, 00:06:12 DEBUG "0" 00:06:12 DEBUG 00:06:12 DEBUG }, 00:06:12 DEBUG 00:06:12 DEBUG "instance": "10.142.0.43:9100" 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "instance": "10.142.0.43:9100" 00:06:12 DEBUG "value": [ 00:06:12 DEBUG }, 00:06:12 DEBUG }, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG "value": [ 00:06:12 DEBUG ] 00:06:12 DEBUG "pod": "ds-cts-2" 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "instance": "10.142.0.43:9100" 00:06:12 DEBUG }, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG }, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "0" 00:06:12 DEBUG "5338.774447025115" 00:06:12 DEBUG "24798554.166666664" 00:06:12 DEBUG "6.350700561021682" 00:06:12 DEBUG "pod": "ds-cts-2" 00:06:12 DEBUG "0" 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG }, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG "0" 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "463.32559166666664" 00:06:12 DEBUG "value": [ 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG "257655997440" 00:06:12 DEBUG "11698.5" 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG }, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG "value": [ 00:06:12 DEBUG ] 00:06:12 DEBUG }, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG ] 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "7573258.546962165" 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG } 00:06:12 DEBUG "value": [ 00:06:12 DEBUG "0.02773333333333312" 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG "0" 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG "0.000027104444444444443" 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG "195933.0222058217" 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG "0" 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG 1713481572, 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG "24798456.2" 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG ] 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:06:12 DEBUG } 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "1016.6666666666666" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "7.275911387107549" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 ERROR [http_cmd]: ERROR 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 ERROR --- status code --- 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 ERROR --- http response --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:07:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:07:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "1016.7" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481632" 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "ds-idrepo-0" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "pod": "ds-idrepo-2" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "0" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "instance": "10.142.0.107:9100" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "instance": "10.142.0.43:9100" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "0.018666666666666797" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713481632" 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "ds-idrepo-0" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "pod": "ds-idrepo-2" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "0" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "4200407040" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "ds-cts-0" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "pod": "ds-cts-2" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "0" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO 00:07:12 INFO [loop_until]: kubectl --namespace=xlou top pods 00:07:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "1016.4333333333333" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "4469.884183475904" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO 00:07:12 INFO [loop_until]: kubectl --namespace=xlou top node 00:07:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG { 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "0" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:07:12 DEBUG } 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "0" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481632" 00:07:12 INFO [loop_until]: OK (rc = 0) 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- stdout --- 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 6014m 4536Mi am-776d56b7dd-m7qb7 4566m 4302Mi am-776d56b7dd-xr8wp 6564m 4591Mi ds-cts-0 2863m 6218Mi ds-cts-1 2860m 5690Mi ds-cts-2 2653m 5648Mi ds-idrepo-0 2692m 3250Mi ds-idrepo-1 2671m 2893Mi ds-idrepo-2 2590m 2989Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1168Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-7dccb59994-hvp6g 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7886m 4336Mi 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- status code --- 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- stderr --- 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG ----- output ----- 00:07:12 INFO [http_cmd]: http status code OK 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG { 00:07:12 DEBUG --- status code --- 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG { 00:07:12 DEBUG { 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG { 00:07:12 DEBUG { 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG --- http response --- 00:07:12 DEBUG { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG "data": { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG { 00:07:12 DEBUG "result": [ 00:07:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "data": { 00:07:12 DEBUG { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "data": { 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG { 00:07:12 DEBUG ----- output ----- 00:07:12 DEBUG "data": { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "data": { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG "result": [ 00:07:12 DEBUG "data": { 00:07:12 DEBUG { 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG "result": [ 00:07:12 DEBUG "result": [ 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:07:12 DEBUG "status": "success", 00:07:12 DEBUG "result": [ 00:07:12 DEBUG { 00:07:12 DEBUG { 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "metric": { 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG { 00:07:12 DEBUG { 00:07:12 DEBUG "result": [ 00:07:12 DEBUG }, 00:07:12 DEBUG "data": { 00:07:12 DEBUG { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "result": [ 00:07:12 DEBUG "instance": "10.142.0.107:9100" 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "ds-cts-0" 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "metric": { 00:07:12 DEBUG { 00:07:12 DEBUG "value": [ 00:07:12 DEBUG "resultType": "vector", 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:07:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:07:12 DEBUG { 00:07:12 DEBUG }, 00:07:12 DEBUG "pod": "ds-cts-0" 00:07:12 DEBUG }, 00:07:12 DEBUG "instance": "10.142.0.107:9100" 00:07:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:07:12 DEBUG "metric": { 00:07:12 DEBUG 00:07:12 DEBUG "result": [ 00:07:12 DEBUG "instance": "10.142.0.107:9100" 00:07:12 DEBUG }, 00:07:12 DEBUG }, 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "value": [ 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG }, 00:07:12 DEBUG }, 00:07:12 DEBUG "instance": "10.142.0.107:9100" 00:07:12 DEBUG 00:07:12 DEBUG { 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG "value": [ 00:07:12 DEBUG "pod": "ds-idrepo-0" 00:07:12 DEBUG 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG "value": [ 00:07:12 DEBUG "value": [ 00:07:12 DEBUG }, 00:07:12 DEBUG 00:07:12 DEBUG "metric": { 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG }, 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "value": [ 00:07:12 DEBUG [...] 00:07:12 DEBUG "pod": "ds-idrepo-0" 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG }, 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG }, 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "instance": "10.142.0.43:9100" 00:07:12 DEBUG 00:07:12 DEBUG "pod": "ds-cts-2" 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG 00:07:12 DEBUG "value": [ 00:07:12 DEBUG [...] 00:07:12 DEBUG 00:07:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:07:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:07:12 DEBUG 00:07:12 DEBUG }, 00:07:12 DEBUG "pod": "ds-cts-2" 00:07:12 DEBUG }, 00:07:12 DEBUG "instance": "10.142.0.43:9100" 00:07:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:07:12 DEBUG 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG 00:07:12 DEBUG "instance": "10.142.0.43:9100" 00:07:12 DEBUG }, 00:07:12 DEBUG }, 00:07:12 DEBUG 00:07:12 DEBUG "value": [ 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG }, 00:07:12 DEBUG }, 00:07:12 DEBUG "instance": "10.142.0.43:9100" 00:07:12 DEBUG "72157.5" 00:07:12 DEBUG 00:07:12 DEBUG }, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG "value": [ 00:07:12 DEBUG "pod": "ds-cts-2" 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG "value": [ 00:07:12 DEBUG }, 00:07:12 DEBUG ] 00:07:12 DEBUG 00:07:12 DEBUG "value": [ 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG }, 00:07:12 DEBUG "28777938.1" 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "0" 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "value": [ 00:07:12 DEBUG } 00:07:12 DEBUG "pod": "ds-idrepo-2" 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG "0" 00:07:12 DEBUG "576.7592494807618" 00:07:12 DEBUG "value": [ 00:07:12 DEBUG ] 00:07:12 DEBUG "0" 00:07:12 DEBUG ] 00:07:12 DEBUG "0.027166666666665642" 00:07:12 DEBUG "0.0029150209455416805" 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG ] 00:07:12 DEBUG }, 00:07:12 DEBUG "258851462144" 00:07:12 DEBUG ] 00:07:12 DEBUG ] 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG ] 00:07:12 DEBUG "28776354.2" 00:07:12 DEBUG } 00:07:12 DEBUG "value": [ 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG "0" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG 1713481632, 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG ] 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG "0" 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG ] 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 DEBUG } 00:07:12 INFO [loop_until]: OK (rc = 0) 00:07:12 DEBUG --- stdout --- 00:07:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6009Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2626m 8% 6648Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 50m 0% 2592Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 5877m 19% 7745Mi 6% gke-xlou-cdm-default-pool-1b0769c8-w15q 2801m 9% 8651Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 5174Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 54m 0% 2586Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 23797m 79% 8540Mi 7% gke-xlou-cdm-ds-0f601751-hmm7 4970m 16% 6429Mi 5% gke-xlou-cdm-ds-0f601751-j488 2731m 9% 6028Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 2802m 9% 9382Mi 8% gke-xlou-cdm-ds-0f601751-nj6q 2957m 9% 6506Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2687m 9% 7552Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8156m 51% 8443Mi 14% 00:07:12 DEBUG --- stderr --- 00:07:12 DEBUG 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "1048.1333333333332" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "1048.1333333333334" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "ds-idrepo-0" 00:08:12 DEBUG }, 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:08:12 DEBUG "value": [ 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG 00:08:12 DEBUG { 00:08:12 DEBUG 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG 00:08:12 DEBUG "data": { 00:08:12 DEBUG [...] 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG 00:08:12 DEBUG "result": [ 00:08:12 DEBUG 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 DEBUG { 00:08:12 DEBUG 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "ds-idrepo-2" 00:08:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:08:12 DEBUG }, 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG 00:08:12 DEBUG "0" 00:08:12 DEBUG 00:08:12 DEBUG ] 00:08:12 DEBUG 00:08:12 DEBUG } 00:08:12 DEBUG [...] 00:08:12 DEBUG ] 00:08:12 DEBUG 00:08:12 DEBUG } 00:08:12 DEBUG 00:08:12 DEBUG } 00:08:12 DEBUG 00:08:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "8.029523765304523" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 ERROR [http_cmd]: ERROR 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 ERROR --- status code --- 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 DEBUG --- status code --- 00:08:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 ERROR --- http response --- 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:08:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:08:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "1047.8666666666666" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "ds-idrepo-0" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "ds-idrepo-2" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "0" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "instance": "10.142.0.107:9100" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "instance": "10.142.0.43:9100" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "0.019999999999999542" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "6307606528" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "ds-cts-0" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "ds-cts-2" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "0" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "2075.029631660726" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "0" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "0" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "133719.5" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO 00:08:12 INFO [loop_until]: kubectl --namespace=xlou top node 00:08:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:08:12 INFO 00:08:12 INFO [loop_until]: kubectl --namespace=xlou top pods 00:08:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG { 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "pod": "ds-idrepo-0" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "ds-cts-2" 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "0" 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481692" 00:08:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481692" 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG --- http response --- 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG --- http response --- 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG { 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG { 00:08:12 DEBUG { 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG "status": "success", 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- http response --- 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:08:12 DEBUG "data": { 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:08:12 DEBUG "data": { 00:08:12 DEBUG "data": { 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG { 00:08:12 DEBUG "result": [ 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG { 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG { 00:08:12 DEBUG "result": [ 00:08:12 DEBUG "result": [ 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG { 00:08:12 INFO [http_cmd]: http status code OK 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG { 00:08:12 DEBUG { 00:08:12 DEBUG "data": { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG --- status code --- 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG "data": { 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG "data": { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "instance": "10.142.0.107:9100" 00:08:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:08:12 DEBUG { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG { 00:08:12 DEBUG { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:08:12 DEBUG "pod": "ds-cts-0" 00:08:12 DEBUG "result": [ 00:08:12 DEBUG }, 00:08:12 DEBUG --- http response --- 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "result": [ 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG "data": { 00:08:12 DEBUG }, 00:08:12 DEBUG }, 00:08:12 DEBUG { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:08:12 DEBUG "data": { 00:08:12 DEBUG { 00:08:12 DEBUG "data": { 00:08:12 DEBUG "result": [ 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "value": [ 00:08:12 DEBUG "value": [ 00:08:12 DEBUG "value": [ 00:08:12 DEBUG "instance": "10.142.0.107:9100" 00:08:12 DEBUG ----- output ----- 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG { 00:08:12 DEBUG "result": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG }, 00:08:12 DEBUG { 00:08:12 DEBUG "result": [ 00:08:12 DEBUG "instance": "10.142.0.107:9100" 00:08:12 DEBUG "result": [ 00:08:12 DEBUG "metric": { 00:08:12 DEBUG { 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "value": [ 00:08:12 DEBUG "status": "success", 00:08:12 DEBUG { 00:08:12 DEBUG }, 00:08:12 DEBUG { 00:08:12 DEBUG "pod": "ds-cts-0" 00:08:12 DEBUG "metric": { 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "data": { 00:08:12 DEBUG "metric": { 00:08:12 DEBUG "value": [ 00:08:12 DEBUG "metric": { 00:08:12 DEBUG }, 00:08:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:08:12 DEBUG [...] 00:08:12 DEBUG [...] 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "resultType": "vector", 00:08:12 DEBUG 00:08:12 DEBUG "pod": "ds-idrepo-0" 00:08:12 DEBUG "value": [ 00:08:12 DEBUG }, 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 INFO [loop_until]: OK (rc = 0) 00:08:12 DEBUG --- stdout --- 00:08:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:08:12 DEBUG [...] 00:08:12 DEBUG "result": [ 00:08:12 DEBUG 00:08:12 DEBUG }, 00:08:12 DEBUG 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG }, 00:08:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 5947m 4738Mi am-776d56b7dd-m7qb7 4601m 4697Mi am-776d56b7dd-xr8wp 6606m 4844Mi ds-cts-0 3007m 6259Mi ds-cts-1 2961m 5704Mi ds-cts-2 2790m 5654Mi ds-idrepo-0 2642m 3250Mi ds-idrepo-1 2633m 2893Mi ds-idrepo-2 2706m 2976Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1169Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-7dccb59994-hvp6g 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8014m 7122Mi 00:08:12 DEBUG 00:08:12 DEBUG { 00:08:12 DEBUG 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "value": [ 00:08:12 DEBUG --- stderr --- 00:08:12 DEBUG 00:08:12 DEBUG "metric": { 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:08:12 DEBUG "pod": "ds-cts-2" 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "instance": "10.142.0.107:9100" 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG 00:08:12 DEBUG "instance": "10.142.0.43:9100" 00:08:12 DEBUG }, 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG }, 00:08:12 DEBUG "instance": "10.142.0.43:9100" 00:08:12 DEBUG "0" 00:08:12 DEBUG 00:08:12 DEBUG }, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG [...] 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG "value": [ 00:08:12 DEBUG }, 00:08:12 DEBUG ] 00:08:12 DEBUG [...] 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG "instance": "10.142.0.43:9100" 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG } 00:08:12 DEBUG 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG 00:08:12 DEBUG }, 00:08:12 DEBUG 00:08:12 DEBUG "pod": "ds-cts-2" 00:08:12 DEBUG 00:08:12 DEBUG "0" 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG ] 00:08:12 DEBUG 00:08:12 DEBUG "260174093312" 00:08:12 DEBUG 00:08:12 DEBUG "value": [ 00:08:12 DEBUG 00:08:12 DEBUG }, 00:08:12 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:08:12 DEBUG ] 00:08:12 DEBUG "30374158.266666666" 00:08:12 DEBUG } 00:08:12 DEBUG 00:08:12 DEBUG ] 00:08:12 DEBUG [...] 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "pod": "ds-idrepo-2" 00:08:12 DEBUG "value": [ 00:08:12 DEBUG }, 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:08:12 DEBUG } 00:08:12 DEBUG 00:08:12 DEBUG "0.02909999999999968" 00:08:12 DEBUG }, 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG "value": [ 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG }, 00:08:12 DEBUG ] 00:08:12 DEBUG 00:08:12 DEBUG ] 00:08:12 DEBUG "value": [ 00:08:12 DEBUG "0.000027104444444444443" 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG "value": [ 00:08:12 DEBUG } 00:08:12 DEBUG 00:08:12 DEBUG } 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG ] 00:08:12 DEBUG "0" 00:08:12 DEBUG ] 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG } 00:08:12 DEBUG "instance": "10.142.0.43:9100" 00:08:12 DEBUG ] 00:08:12 DEBUG "0" 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG "0.006400574799378486" 00:08:12 DEBUG }, 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG "value": [ 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG 1713481692, 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG "30372348.966666665" 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 DEBUG ] 00:08:12 DEBUG } 00:08:12 DEBUG } 00:08:12 INFO [loop_until]: OK (rc = 0) 00:08:12 DEBUG --- stdout --- 00:08:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6009Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2726m 9% 6635Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 48m 0% 2589Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 5770m 19% 7909Mi 7% gke-xlou-cdm-default-pool-1b0769c8-w15q 3060m 10% 8658Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 5204Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2588Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 23962m 80% 8797Mi 7% gke-xlou-cdm-ds-0f601751-hmm7 5053m 16% 6801Mi 6% gke-xlou-cdm-ds-0f601751-j488 2827m 9% 6016Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 3006m 10% 9368Mi 8% gke-xlou-cdm-ds-0f601751-nj6q 2955m 9% 6511Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2799m 9% 7525Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8208m 51% 10525Mi 17% 00:08:12 DEBUG --- stderr --- 00:08:12 DEBUG 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "ds-idrepo-0" 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:09:12 DEBUG }, 00:09:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "pod": "ds-idrepo-2" 00:09:12 DEBUG 00:09:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:09:12 DEBUG }, 00:09:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "0" 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "1018.4666666666667" 00:09:12 DEBUG ] 00:09:12 DEBUG "1018.4666666666667" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG { 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG "data": { 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:09:12 DEBUG 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG 00:09:12 DEBUG { 00:09:12 DEBUG [...] 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG 00:09:12 DEBUG "data": { 00:09:12 DEBUG 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG 00:09:12 DEBUG "result": [ 00:09:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:09:12 DEBUG { 00:09:12 DEBUG }, 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "value": [ 00:09:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG }, 00:09:12 DEBUG "1018.4333333333333" 00:09:12 DEBUG "value": [ 00:09:12 DEBUG ] 00:09:12 DEBUG 00:09:12 DEBUG } 00:09:12 DEBUG 00:09:12 DEBUG ] 00:09:12 DEBUG 00:09:12 DEBUG } 00:09:12 DEBUG [...] 00:09:12 DEBUG } 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "8.041542861990528" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "instance": "10.142.0.107:9100" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "instance": "10.142.0.43:9100" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "0.021666666666666504" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713481752" 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "ds-idrepo-0" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "pod": "ds-idrepo-2" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "0" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 ERROR [http_cmd]: ERROR 00:09:12 ERROR --- status code --- 00:09:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:09:12 ERROR --- http response --- 00:09:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:09:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "8322023424" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "0" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "ds-cts-0" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "pod": "ds-cts-2" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "0" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "0" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "ds-idrepo-0" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "pod": "ds-cts-2" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "0" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "2699.0235704535116" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481752" 00:09:12 INFO [http_cmd]: http status code OK 00:09:12 DEBUG --- status code --- 00:09:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:12 DEBUG --- http response --- 00:09:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:09:12 DEBUG ----- output ----- 00:09:12 DEBUG { 00:09:12 DEBUG "status": "success", 00:09:12 DEBUG "data": { 00:09:12 DEBUG "resultType": "vector", 00:09:12 DEBUG "result": [ 00:09:12 DEBUG { 00:09:12 DEBUG "metric": { 00:09:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG [...] 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG 00:09:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:09:12 DEBUG }, 00:09:12 DEBUG "value": [ 00:09:12 DEBUG 1713481752, 00:09:12 DEBUG "197473" 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG ] 00:09:12 DEBUG } 00:09:12 DEBUG } 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481753" 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 DEBUG --- status code --- 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 DEBUG --- http response --- 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:09:13 DEBUG ----- output ----- 00:09:13 DEBUG { 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG "data": { 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG "result": [ 00:09:13 DEBUG { 00:09:13 DEBUG "metric": { 00:09:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:09:13 DEBUG }, 00:09:13 DEBUG "value": [ 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG [...] 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:09:13 DEBUG }, 00:09:13 DEBUG "value": [ 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG "0" 00:09:13 DEBUG ] 00:09:13 DEBUG } 00:09:13 DEBUG ] 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 INFO 00:09:13 INFO [loop_until]: kubectl --namespace=xlou top pods 00:09:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481753" 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481753" 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481753" 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481753" 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481753" 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481753" 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481753" 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481753" 00:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481753" 00:09:13 INFO 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 INFO [loop_until]: kubectl --namespace=xlou top node 00:09:13 DEBUG --- status code --- 00:09:13 DEBUG --- status code --- 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 DEBUG --- status code --- 00:09:13 DEBUG --- http response --- 00:09:13 DEBUG --- http response --- 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 DEBUG --- http response --- 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 DEBUG ----- output ----- 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 DEBUG ----- output ----- 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 DEBUG --- status code --- 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:09:13 DEBUG --- status code --- 00:09:13 DEBUG --- status code --- 00:09:13 DEBUG { 00:09:13 DEBUG --- status code --- 00:09:13 DEBUG { 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 DEBUG ----- output ----- 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 INFO [http_cmd]: http status code OK 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 DEBUG --- status code --- 00:09:13 DEBUG "data": { 00:09:13 DEBUG --- http response --- 00:09:13 DEBUG { 00:09:13 DEBUG --- http response --- 00:09:13 DEBUG --- http response --- 00:09:13 DEBUG --- status code --- 00:09:13 DEBUG "data": { 00:09:13 DEBUG --- http response --- 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG --- http response --- 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:09:13 DEBUG "result": [ 00:09:13 DEBUG "data": { 00:09:13 DEBUG ----- output ----- 00:09:13 DEBUG ----- output ----- 00:09:13 DEBUG --- http response --- 00:09:13 DEBUG ----- output ----- 00:09:13 DEBUG "result": [ 00:09:13 DEBUG ----- output ----- 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:09:13 DEBUG { 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG { 00:09:13 DEBUG { 00:09:13 DEBUG { 00:09:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:09:13 DEBUG { 00:09:13 DEBUG { 00:09:13 DEBUG ----- output ----- 00:09:13 DEBUG "metric": { 00:09:13 DEBUG "result": [ 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG ----- output ----- 00:09:13 DEBUG "metric": { 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG { 00:09:13 DEBUG "instance": "10.142.0.107:9100" 00:09:13 DEBUG { 00:09:13 DEBUG "data": { 00:09:13 DEBUG "data": { 00:09:13 DEBUG "data": { 00:09:13 DEBUG { 00:09:13 DEBUG "pod": "ds-cts-0" 00:09:13 DEBUG "data": { 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG }, 00:09:13 DEBUG "metric": { 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG "status": "success", 00:09:13 DEBUG }, 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG "data": { 00:09:13 DEBUG "value": [ 00:09:13 DEBUG "instance": "10.142.0.107:9100" 00:09:13 DEBUG "result": [ 00:09:13 DEBUG "result": [ 00:09:13 DEBUG "result": [ 00:09:13 DEBUG "data": { 00:09:13 DEBUG "value": [ 00:09:13 DEBUG "result": [ 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG 00:09:13 DEBUG }, 00:09:13 DEBUG { 00:09:13 DEBUG { 00:09:13 DEBUG { 00:09:13 DEBUG "resultType": "vector", 00:09:13 DEBUG 00:09:13 DEBUG { 00:09:13 DEBUG "result": [ 00:09:13 DEBUG 00:09:13 DEBUG "value": [ 00:09:13 DEBUG "metric": { 00:09:13 DEBUG "metric": { 00:09:13 DEBUG "metric": { 00:09:13 DEBUG "result": [ 00:09:13 DEBUG 00:09:13 DEBUG "metric": { 00:09:13 DEBUG { 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "instance": "10.142.0.107:9100" 00:09:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:09:13 DEBUG "pod": "ds-cts-0" 00:09:13 DEBUG { 00:09:13 DEBUG 00:09:13 DEBUG "instance": "10.142.0.107:9100" 00:09:13 DEBUG "metric": { 00:09:13 DEBUG [...] 00:09:13 DEBUG 00:09:13 DEBUG }, 00:09:13 DEBUG }, 00:09:13 DEBUG }, 00:09:13 DEBUG "metric": { 00:09:13 DEBUG [...] 00:09:13 DEBUG }, 00:09:13 DEBUG "pod": "ds-idrepo-0" 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "value": [ 00:09:13 DEBUG "value": [ 00:09:13 DEBUG "value": [ 00:09:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:09:13 DEBUG 00:09:13 DEBUG "value": [ 00:09:13 DEBUG }, 00:09:13 DEBUG 00:09:13 DEBUG [...] 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG }, 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "value": [ 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "value": [ 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "instance": "10.142.0.43:9100" 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "pod": "ds-cts-2" 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG }, 00:09:13 DEBUG 00:09:13 DEBUG [...] 00:09:13 DEBUG [...] 00:09:13 DEBUG [...] 00:09:13 DEBUG 00:09:13 DEBUG }, 00:09:13 DEBUG [...] 00:09:13 DEBUG 00:09:13 DEBUG "value": [ 00:09:13 DEBUG "instance": "10.142.0.43:9100" 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "value": [ 00:09:13 DEBUG 00:09:13 DEBUG [...] 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG }, 00:09:13 DEBUG "value": [ 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG [...] 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "28850798.233333334" 00:09:13 DEBUG 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG "0" 00:09:13 DEBUG 00:09:13 DEBUG 00:09:13 DEBUG ] 00:09:13 DEBUG 00:09:13 DEBUG "261445291008" 00:09:13 DEBUG "instance": "10.142.0.43:9100" 00:09:13 DEBUG "pod": "ds-cts-2" 00:09:13 DEBUG 00:09:13 DEBUG ] 00:09:13 DEBUG "instance": "10.142.0.43:9100" 00:09:13 DEBUG 00:09:13 DEBUG } 00:09:13 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:09:13 DEBUG ] 00:09:13 DEBUG }, 00:09:13 DEBUG }, 00:09:13 DEBUG 00:09:13 DEBUG } 00:09:13 DEBUG }, 00:09:13 DEBUG "pod": "ds-idrepo-2" 00:09:13 DEBUG ] 00:09:13 DEBUG }, 00:09:13 DEBUG } 00:09:13 DEBUG "value": [ 00:09:13 DEBUG "value": [ 00:09:13 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:09:13 DEBUG ] 00:09:13 DEBUG "value": [ 00:09:13 DEBUG }, 00:09:13 DEBUG } 00:09:13 DEBUG "value": [ 00:09:13 DEBUG ] 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG }, 00:09:13 DEBUG } 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG "value": [ 00:09:13 DEBUG } 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG } 00:09:13 DEBUG "28851208.233333334" 00:09:13 DEBUG "0" 00:09:13 DEBUG "value": [ 00:09:13 DEBUG } 00:09:13 DEBUG "0.029666666666667157" 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG "0" 00:09:13 DEBUG } 00:09:13 DEBUG ] 00:09:13 DEBUG ] 00:09:13 DEBUG 1713481753, 00:09:13 DEBUG ] 00:09:13 DEBUG "0" 00:09:13 DEBUG ] 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG "0.000889963491581991" 00:09:13 DEBUG } 00:09:13 DEBUG ] 00:09:13 DEBUG } 00:09:13 DEBUG ] 00:09:13 DEBUG ] 00:09:13 DEBUG ] 00:09:13 DEBUG ] 00:09:13 DEBUG } 00:09:13 DEBUG ] 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG ] 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG ] 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 DEBUG } 00:09:13 INFO [loop_until]: OK (rc = 0) 00:09:13 DEBUG --- stdout --- 00:09:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 6136m 5710Mi am-776d56b7dd-m7qb7 4709m 4702Mi am-776d56b7dd-xr8wp 6881m 4893Mi ds-cts-0 3102m 6262Mi ds-cts-1 3199m 5741Mi ds-cts-2 2927m 5693Mi ds-idrepo-0 2831m 3250Mi ds-idrepo-1 2625m 2906Mi ds-idrepo-2 2634m 2986Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1169Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-7dccb59994-hvp6g 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8030m 7858Mi 00:09:13 DEBUG --- stderr --- 00:09:13 DEBUG 00:09:13 INFO [loop_until]: OK (rc = 0) 00:09:13 DEBUG --- stdout --- 00:09:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 5998Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2690m 9% 6627Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 49m 0% 2584Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 6288m 21% 9938Mi 8% gke-xlou-cdm-default-pool-1b0769c8-w15q 3011m 10% 8641Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 135m 0% 5190Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 53m 0% 2597Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 24495m 82% 8837Mi 7% gke-xlou-cdm-ds-0f601751-hmm7 5242m 17% 6810Mi 6% gke-xlou-cdm-ds-0f601751-j488 2924m 9% 6031Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 3047m 10% 9383Mi 8% gke-xlou-cdm-ds-0f601751-nj6q 3055m 10% 6520Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2915m 9% 7535Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8240m 51% 11184Mi 19% 00:09:13 DEBUG --- stderr --- 00:09:13 DEBUG 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "956.2666666666667" 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:10:12 DEBUG "pod": "ds-idrepo-0" 00:10:12 DEBUG }, 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:10:12 DEBUG "pod": "ds-idrepo-2" 00:10:12 DEBUG }, 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "956.6" 00:10:12 DEBUG "0" 00:10:12 DEBUG ] 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 DEBUG ] 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713481812" 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713481812" 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "956.2666666666667" 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 ERROR [http_cmd]: ERROR 00:10:12 ERROR --- status code --- 00:10:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:10:12 ERROR --- http response --- 00:10:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:10:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481812" 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "8.026067127066826" 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "instance": "10.142.0.107:9100" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "instance": "10.142.0.43:9100" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "0.0216666666666668" 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "pod": "ds-idrepo-0" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "pod": "ds-idrepo-2" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "0" 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "pod": "ds-idrepo-0" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "pod": "ds-cts-2" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "0" 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713481812" 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "0" 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG ] 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG } 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG } 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "8319803392" 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: http status code OK 00:10:12 DEBUG --- status code --- 00:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:12 DEBUG --- http response --- 00:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:10:12 DEBUG ----- output ----- 00:10:12 DEBUG { 00:10:12 DEBUG "status": "success", 00:10:12 DEBUG "data": { 00:10:12 DEBUG "resultType": "vector", 00:10:12 DEBUG "result": [ 00:10:12 DEBUG { 00:10:12 DEBUG "metric": { 00:10:12 DEBUG "pod": "ds-cts-0" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG [...] 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG 00:10:12 DEBUG "pod": "ds-cts-2" 00:10:12 DEBUG }, 00:10:12 DEBUG "value": [ 00:10:12 DEBUG 1713481812, 00:10:12 DEBUG "0" 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG ] 00:10:12 DEBUG } 00:10:12 DEBUG } 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481812" 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG { 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG "data": { 00:10:13 DEBUG "resultType": "vector", 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG "result": [ 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG { 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG "metric": { 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG [...] 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 1713481812, 00:10:13 DEBUG "0" 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG { 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG "data": { 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG "result": [ 00:10:13 DEBUG { 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG [...] 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 1713481812, 00:10:13 DEBUG "0" 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481813" 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG { 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG "data": { 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG "result": [ 00:10:13 DEBUG { 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG [...] 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG "256554.5" 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481813" 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG { 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG "data": { 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG "result": [ 00:10:13 DEBUG { 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG [...] 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG "0" 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481813" 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG { 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG "data": { 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG "result": [ 00:10:13 DEBUG { 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG [...] 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG "532.709064897906" 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481813" 00:10:13 INFO 00:10:13 INFO [loop_until]: kubectl --namespace=xlou top pods 00:10:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:10:13 INFO 00:10:13 INFO [loop_until]: kubectl --namespace=xlou top node 00:10:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481813" 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG { 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG "data": { 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG "result": [ 00:10:13 DEBUG { 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "instance": "10.142.0.107:9100" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG [...] 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG "instance": "10.142.0.43:9100" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG "27661348.2" 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481813" 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481813" 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481813" 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481813" 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481813" 00:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481813" 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG { 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG "data": { 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG "result": [ 00:10:13 DEBUG { 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "pod": "ds-idrepo-0" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG [...] 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG "pod": "ds-idrepo-2" 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 1713481813, 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG "0" 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG ] 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG } 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG ] 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG } 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:10:13 DEBUG } 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG { 00:10:13 DEBUG { 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG "data": { 00:10:13 DEBUG "data": { 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG "result": [ 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG "result": [ 00:10:13 DEBUG { 00:10:13 DEBUG { 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG { 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "pod": "ds-cts-0" 00:10:13 DEBUG "data": { 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:13 DEBUG "instance": "10.142.0.107:9100" 00:10:13 DEBUG }, 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG "result": [ 00:10:13 DEBUG { 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 00:10:13 INFO [http_cmd]: http status code OK 00:10:13 DEBUG { 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG --- status code --- 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "data": { 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:10:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG 00:10:13 DEBUG [...] 00:10:13 DEBUG --- http response --- 00:10:13 DEBUG }, 00:10:13 DEBUG "result": [ 00:10:13 DEBUG [...] 00:10:13 DEBUG 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:10:13 DEBUG "value": [ 00:10:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:10:13 DEBUG { 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG 00:10:13 DEBUG ----- output ----- 00:10:13 DEBUG "metric": { 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG { 00:10:13 DEBUG 00:10:13 DEBUG { 00:10:13 DEBUG "instance": "10.142.0.107:9100" 00:10:13 DEBUG 00:10:13 DEBUG "instance": "10.142.0.43:9100" 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG 00:10:13 DEBUG "status": "success", 00:10:13 DEBUG }, 00:10:13 DEBUG "pod": "ds-cts-2" 00:10:13 DEBUG }, 00:10:13 DEBUG "data": { 00:10:13 DEBUG [...] 00:10:13 DEBUG "data": { 00:10:13 DEBUG "value": [ 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG 00:10:13 DEBUG "resultType": "vector", 00:10:13 DEBUG 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG "result": [ 00:10:13 DEBUG 00:10:13 DEBUG "result": [ 00:10:13 DEBUG 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG "262659425280" 00:10:13 DEBUG { 00:10:13 DEBUG 00:10:13 DEBUG { 00:10:13 DEBUG 00:10:13 DEBUG "0" 00:10:13 DEBUG ] 00:10:13 DEBUG "metric": { 00:10:13 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:10:13 DEBUG "metric": { 00:10:13 DEBUG [...] 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG "pod": "ds-cts-0" 00:10:13 DEBUG }, 00:10:13 DEBUG "instance": "10.142.0.107:9100" 00:10:13 DEBUG 00:10:13 DEBUG } 00:10:13 DEBUG ] 00:10:13 DEBUG }, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG }, 00:10:13 DEBUG 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG "value": [ 00:10:13 DEBUG 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 DEBUG 00:10:13 DEBUG "0" 00:10:13 DEBUG 00:10:13 DEBUG "instance": "10.142.0.43:9100" 00:10:13 DEBUG } 00:10:13 DEBUG 00:10:13 DEBUG ] 00:10:13 DEBUG 00:10:13 DEBUG }, 00:10:13 DEBUG 00:10:13 DEBUG } 00:10:13 DEBUG 00:10:13 DEBUG "value": [ 00:10:13 DEBUG [...] 00:10:13 DEBUG ] 00:10:13 DEBUG [...] 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG 00:10:13 DEBUG } 00:10:13 DEBUG 00:10:13 DEBUG "0.029500000000000644" 00:10:13 DEBUG 00:10:13 DEBUG } 00:10:13 DEBUG 00:10:13 DEBUG ] 00:10:13 DEBUG 00:10:13 DEBUG 00:10:13 DEBUG } 00:10:13 DEBUG "pod": "ds-cts-2" 00:10:13 DEBUG "instance": "10.142.0.43:9100" 00:10:13 DEBUG ] 00:10:13 DEBUG }, 00:10:13 DEBUG }, 00:10:13 DEBUG } 00:10:13 DEBUG "value": [ 00:10:13 DEBUG "value": [ 00:10:13 DEBUG } 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG 1713481813, 00:10:13 DEBUG "0" 00:10:13 DEBUG "27659686.933333334" 00:10:13 DEBUG ] 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 DEBUG ] 00:10:13 DEBUG ] 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 DEBUG } 00:10:13 INFO [loop_until]: OK (rc = 0) 00:10:13 DEBUG --- stdout --- 00:10:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 5530m 7126Mi am-776d56b7dd-m7qb7 4533m 6853Mi am-776d56b7dd-xr8wp 6275m 4894Mi ds-cts-0 2728m 6300Mi ds-cts-1 2855m 5776Mi ds-cts-2 2649m 5725Mi ds-idrepo-0 2472m 3250Mi ds-idrepo-1 2553m 2906Mi ds-idrepo-2 2656m 2986Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1168Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-7dccb59994-hvp6g 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8067m 7941Mi 00:10:13 DEBUG --- stderr --- 00:10:13 DEBUG 00:10:13 INFO [loop_until]: OK (rc = 0) 00:10:13 DEBUG --- stdout --- 00:10:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6007Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2562m 8% 6633Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 2579Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 5399m 18% 10287Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 2784m 9% 8671Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5183Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2592Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 23342m 78% 8868Mi 7% gke-xlou-cdm-ds-0f601751-hmm7 5019m 16% 8934Mi 8% gke-xlou-cdm-ds-0f601751-j488 2580m 8% 6027Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 2930m 9% 9410Mi 8% gke-xlou-cdm-ds-0f601751-nj6q 2789m 9% 6526Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2660m 8% 7559Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8299m 52% 10906Mi 18% 00:10:13 DEBUG --- stderr --- 00:10:13 DEBUG 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481872" 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481872" 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481872" 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713481872" 00:11:12 INFO [http_cmd]: http status code OK 00:11:12 DEBUG --- status code --- 00:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:12 DEBUG --- http response --- 00:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:11:12 DEBUG ----- output ----- 00:11:12 DEBUG { 00:11:12 DEBUG "status": "success", 00:11:12 DEBUG "data": { 00:11:12 DEBUG "resultType": "vector", 00:11:12 DEBUG "result": [ 00:11:12 DEBUG { 00:11:12 DEBUG "metric": { 00:11:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG [...] 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 1713481872, 00:11:12 DEBUG "393.76666666666665" 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG } 00:11:12 INFO [http_cmd]: http status code OK 00:11:12 INFO [http_cmd]: http status code OK 00:11:12 DEBUG --- status code --- 00:11:12 DEBUG --- status code --- 00:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:12 DEBUG --- http response --- 00:11:12 DEBUG --- http response --- 00:11:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:11:12 DEBUG ----- output ----- 00:11:12 DEBUG { 00:11:12 DEBUG "status": "success", 00:11:12 DEBUG "data": { 00:11:12 DEBUG "resultType": "vector", 00:11:12 DEBUG "result": [ 00:11:12 DEBUG { 00:11:12 DEBUG "metric": { 00:11:12 DEBUG "pod": "ds-idrepo-0" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG [...] 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG "pod": "ds-idrepo-2" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 1713481872, 00:11:12 DEBUG "0" 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG } 00:11:12 INFO [http_cmd]: http status code OK 00:11:12 DEBUG --- status code --- 00:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:12 DEBUG --- http response --- 00:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:11:12 DEBUG ----- output ----- 00:11:12 DEBUG { 00:11:12 DEBUG "status": "success", 00:11:12 DEBUG "data": { 00:11:12 DEBUG "resultType": "vector", 00:11:12 DEBUG "result": [ 00:11:12 DEBUG { 00:11:12 DEBUG "metric": { 00:11:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG [...] 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 1713481872, 00:11:12 DEBUG "8.026497108722067" 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG } 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481872" 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481872" 00:11:12 INFO [http_cmd]: http status code OK 00:11:12 DEBUG --- status code --- 00:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:12 DEBUG --- http response --- 00:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:11:12 DEBUG ----- output ----- 00:11:12 DEBUG { 00:11:12 DEBUG "status": "success", 00:11:12 DEBUG "data": { 00:11:12 DEBUG "resultType": "vector", 00:11:12 DEBUG "result": [ 00:11:12 DEBUG { 00:11:12 DEBUG "metric": { 00:11:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG [...] 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 1713481872, 00:11:12 DEBUG "394.23333333333335" 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG } 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713481872" 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481872" 00:11:12 INFO [http_cmd]: http status code OK 00:11:12 DEBUG --- status code --- 00:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:12 DEBUG --- http response --- 00:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:11:12 DEBUG ----- output ----- 00:11:12 DEBUG { 00:11:12 DEBUG "status": "success", 00:11:12 DEBUG "data": { 00:11:12 DEBUG "resultType": "vector", 00:11:12 DEBUG "result": [ 00:11:12 DEBUG { 00:11:12 DEBUG "metric": { 00:11:12 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG [...] 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 1713481872, 00:11:12 DEBUG "393.7" 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG } 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481872" 00:11:12 ERROR [http_cmd]: ERROR 00:11:12 ERROR --- status code --- 00:11:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:11:12 ERROR --- http response --- 00:11:12 INFO [http_cmd]: http status code OK 00:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481872" 00:11:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:11:12 DEBUG --- status code --- 00:11:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:12 DEBUG --- http response --- 00:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:11:12 DEBUG ----- output ----- 00:11:12 DEBUG { 00:11:12 DEBUG "status": "success", 00:11:12 DEBUG "data": { 00:11:12 DEBUG "resultType": "vector", 00:11:12 DEBUG "result": [ 00:11:12 DEBUG { 00:11:12 DEBUG "metric": { 00:11:12 DEBUG "pod": "ds-idrepo-0" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG [...] 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG 00:11:12 DEBUG "pod": "ds-idrepo-2" 00:11:12 DEBUG }, 00:11:12 DEBUG "value": [ 00:11:12 DEBUG 1713481872, 00:11:12 DEBUG "0" 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG ] 00:11:12 DEBUG } 00:11:12 DEBUG } 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "instance": "10.142.0.107:9100" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "instance": "10.142.0.43:9100" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481872, 00:11:13 DEBUG "0.006333333333333553" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:13 DEBUG }, 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:11:13 DEBUG "value": [ 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG 00:11:13 DEBUG { 00:11:13 DEBUG 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG 00:11:13 DEBUG "data": { 00:11:13 DEBUG [...] 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG 00:11:13 DEBUG "result": [ 00:11:13 DEBUG 00:11:13 DEBUG { 00:11:13 DEBUG 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:11:13 DEBUG "pod": "ds-idrepo-0" 00:11:13 DEBUG }, 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG 00:11:13 DEBUG "0" 00:11:13 DEBUG 00:11:13 DEBUG ] 00:11:13 DEBUG 00:11:13 DEBUG } 00:11:13 DEBUG [...] 00:11:13 DEBUG ] 00:11:13 DEBUG 00:11:13 DEBUG } 00:11:13 DEBUG 00:11:13 DEBUG } 00:11:13 DEBUG 00:11:13 DEBUG "pod": "ds-cts-2" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "0" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713481873" 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "pod": "ds-cts-0" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "pod": "ds-cts-2" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "0" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG { 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG "data": { 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "data": { 00:11:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG }, 00:11:13 DEBUG "result": [ 00:11:13 DEBUG "value": [ 00:11:13 DEBUG { 00:11:13 DEBUG 00:11:13 DEBUG "metric": { 00:11:13 DEBUG 00:11:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:11:13 DEBUG 00:11:13 DEBUG }, 00:11:13 DEBUG [...] 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:11:13 DEBUG [...] 00:11:13 DEBUG }, 00:11:13 DEBUG 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG 00:11:13 DEBUG "0" 00:11:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:11:13 DEBUG ] 00:11:13 DEBUG }, 00:11:13 DEBUG } 00:11:13 DEBUG "value": [ 00:11:13 DEBUG ] 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG } 00:11:13 DEBUG "8537513984" 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "307137.5" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "319989.31111649313" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "0" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "0" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO 00:11:13 INFO [loop_until]: kubectl --namespace=xlou top pods 00:11:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481873" 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "instance": "10.142.0.107:9100" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "instance": "10.142.0.43:9100" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "9320655.6" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481873" 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481873" 00:11:13 DEBUG --- status code --- 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481873" 00:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481873" 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "pod": "ds-idrepo-0" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "pod": "ds-idrepo-2" 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "0" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG { 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG "data": { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "data": { 00:11:13 DEBUG { 00:11:13 DEBUG "result": [ 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "metric": { 00:11:13 DEBUG { 00:11:13 DEBUG "result": [ 00:11:13 DEBUG "pod": "ds-cts-0" 00:11:13 DEBUG "metric": { 00:11:13 DEBUG { 00:11:13 DEBUG }, 00:11:13 DEBUG "instance": "10.142.0.107:9100" 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "value": [ 00:11:13 DEBUG }, 00:11:13 DEBUG "instance": "10.142.0.107:9100" 00:11:13 DEBUG 00:11:13 DEBUG "value": [ 00:11:13 DEBUG }, 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "pod": "ds-cts-2" 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG }, 00:11:13 DEBUG "instance": "10.142.0.43:9100" 00:11:13 DEBUG 00:11:13 DEBUG "value": [ 00:11:13 DEBUG }, 00:11:13 DEBUG "instance": "10.142.0.43:9100" 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG }, 00:11:13 DEBUG "0" 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG ] 00:11:13 DEBUG "263688272896" 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG "0.010200000000001333" 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:11:13 INFO [http_cmd]: http status code OK 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG --- status code --- 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG { 00:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG --- http response --- 00:11:13 DEBUG "data": { 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG "result": [ 00:11:13 DEBUG ----- output ----- 00:11:13 DEBUG { 00:11:13 DEBUG { 00:11:13 DEBUG { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "status": "success", 00:11:13 DEBUG "data": { 00:11:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:11:13 DEBUG "data": { 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG }, 00:11:13 DEBUG "resultType": "vector", 00:11:13 DEBUG "result": [ 00:11:13 DEBUG "value": [ 00:11:13 DEBUG "result": [ 00:11:13 DEBUG { 00:11:13 DEBUG 00:11:13 DEBUG { 00:11:13 DEBUG "metric": { 00:11:13 DEBUG 00:11:13 DEBUG "metric": { 00:11:13 DEBUG "pod": "ds-cts-0" 00:11:13 DEBUG 00:11:13 DEBUG "instance": "10.142.0.107:9100" 00:11:13 DEBUG }, 00:11:13 DEBUG [...] 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:11:13 DEBUG 00:11:13 DEBUG [...] 00:11:13 DEBUG }, 00:11:13 DEBUG [...] 00:11:13 DEBUG 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG 00:11:13 DEBUG 00:11:13 DEBUG "3.6104283194720255" 00:11:13 DEBUG 00:11:13 DEBUG "pod": "ds-cts-2" 00:11:13 DEBUG ] 00:11:13 DEBUG "instance": "10.142.0.43:9100" 00:11:13 DEBUG }, 00:11:13 DEBUG } 00:11:13 DEBUG }, 00:11:13 DEBUG "value": [ 00:11:13 DEBUG ] 00:11:13 DEBUG "value": [ 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG } 00:11:13 DEBUG 1713481873, 00:11:13 DEBUG "0" 00:11:13 DEBUG } 00:11:13 DEBUG "9319099.933333334" 00:11:13 DEBUG ] 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 DEBUG ] 00:11:13 DEBUG ] 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 DEBUG } 00:11:13 INFO [loop_until]: OK (rc = 0) 00:11:13 DEBUG --- stdout --- 00:11:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 5464m 7157Mi am-776d56b7dd-m7qb7 3331m 7195Mi am-776d56b7dd-xr8wp 5612m 5990Mi ds-cts-0 2789m 6332Mi ds-cts-1 2425m 5848Mi ds-cts-2 2362m 6015Mi ds-idrepo-0 2393m 3249Mi ds-idrepo-1 2520m 2909Mi ds-idrepo-2 2453m 2987Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1168Mi idm-b85f9d978-wnhxb 4m 1214Mi lodemon-7dccb59994-hvp6g 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7039m 15Mi 00:11:13 DEBUG --- stderr --- 00:11:13 DEBUG 00:11:13 INFO 00:11:13 INFO [loop_until]: kubectl --namespace=xlou top node 00:11:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:11:13 INFO [loop_until]: OK (rc = 0) 00:11:13 DEBUG --- stdout --- 00:11:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6002Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2314m 7% 6628Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 50m 0% 2585Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 4906m 16% 10331Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 2857m 9% 8685Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 5197Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2601Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 18189m 60% 9212Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 4050m 13% 9305Mi 8% gke-xlou-cdm-ds-0f601751-j488 2659m 8% 6042Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 2367m 7% 9468Mi 8% gke-xlou-cdm-ds-0f601751-nj6q 2336m 7% 6519Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2575m 8% 7841Mi 7% gke-xlou-cdm-frontend-a8771548-i82n 6275m 39% 3872Mi 6% 00:11:13 DEBUG --- stderr --- 00:11:13 DEBUG 00:11:15 WARNING File /tmp/lodemon_data-20240419_000409/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [19/Apr/2024 00:11:15] "GET /monitoring/average?start_time=20240418_230716&stop_time=20240418_230915 HTTP/1.1" 200 - 00:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481932" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481932, 00:12:13 DEBUG "842.2333333333333" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0.5469841843606471" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "ds-idrepo-0" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "ds-idrepo-2" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "ds-idrepo-0" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "ds-idrepo-2" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "instance": "10.142.0.107:9100" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "instance": "10.142.0.43:9100" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0.01999999999999969" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "841.9" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "842.2666666666667" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 ERROR [http_cmd]: ERROR 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 ERROR --- status code --- 00:12:13 DEBUG --- status code --- 00:12:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 ERROR --- http response --- 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:12:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "ds-idrepo-0" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "ds-cts-2" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "450695168" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "ds-cts-0" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "ds-cts-2" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:12:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:12:13 DEBUG }, 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:12:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:12:13 DEBUG }, 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG "325701" 00:12:13 DEBUG ] 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "ds-idrepo-0" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "ds-idrepo-0" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "3807.3015160742325" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "instance": "10.142.0.107:9100" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "instance": "10.142.0.43:9100" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "26261240.03333333" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "ds-cts-0" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "ds-cts-2" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "ds-idrepo-0" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481933" 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "ds-idrepo-2" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481933" 00:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481933" 00:12:13 INFO 00:12:13 INFO [loop_until]: kubectl --namespace=xlou top pods 00:12:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "instance": "10.142.0.107:9100" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "instance": "10.142.0.43:9100" 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0.02586666666666607" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG --- status code --- 00:12:13 INFO [http_cmd]: http status code OK 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- status code --- 00:12:13 DEBUG { 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG --- http response --- 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:12:13 DEBUG "data": { 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG ----- output ----- 00:12:13 DEBUG { 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG { 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "status": "success", 00:12:13 DEBUG "data": { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "data": { 00:12:13 DEBUG "data": { 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "instance": "10.142.0.107:9100" 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "resultType": "vector", 00:12:13 DEBUG "result": [ 00:12:13 DEBUG }, 00:12:13 DEBUG "result": [ 00:12:13 DEBUG "result": [ 00:12:13 DEBUG { 00:12:13 DEBUG "value": [ 00:12:13 DEBUG { 00:12:13 DEBUG { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "metric": { 00:12:13 DEBUG "pod": "ds-cts-0" 00:12:13 DEBUG 00:12:13 DEBUG "instance": "10.142.0.107:9100" 00:12:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:12:13 DEBUG }, 00:12:13 DEBUG 00:12:13 DEBUG }, 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG [...] 00:12:13 DEBUG "value": [ 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG [...] 00:12:13 DEBUG "instance": "10.142.0.43:9100" 00:12:13 DEBUG [...] 00:12:13 DEBUG [...] 00:12:13 DEBUG 00:12:13 DEBUG }, 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG 00:12:13 DEBUG 00:12:13 DEBUG "pod": "ds-cts-2" 00:12:13 DEBUG "264048491520" 00:12:13 DEBUG "instance": "10.142.0.43:9100" 00:12:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:12:13 DEBUG }, 00:12:13 DEBUG ] 00:12:13 DEBUG }, 00:12:13 DEBUG }, 00:12:13 DEBUG "value": [ 00:12:13 DEBUG } 00:12:13 DEBUG "value": [ 00:12:13 DEBUG "value": [ 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG ] 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG 1713481933, 00:12:13 DEBUG "0.00002766" 00:12:13 DEBUG } 00:12:13 DEBUG "26259761.46666667" 00:12:13 DEBUG "0" 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 DEBUG ] 00:12:13 DEBUG ] 00:12:13 DEBUG ] 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 DEBUG } 00:12:13 INFO [loop_until]: OK (rc = 0) 00:12:13 DEBUG --- stdout --- 00:12:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 354m 7155Mi am-776d56b7dd-m7qb7 529m 7196Mi am-776d56b7dd-xr8wp 1351m 5986Mi ds-cts-0 581m 6336Mi ds-cts-1 826m 5977Mi ds-cts-2 224m 6097Mi ds-idrepo-0 812m 3250Mi ds-idrepo-1 360m 2909Mi ds-idrepo-2 153m 2986Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1168Mi idm-b85f9d978-wnhxb 4m 1214Mi lodemon-7dccb59994-hvp6g 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 3261m 434Mi 00:12:13 DEBUG --- stderr --- 00:12:13 DEBUG 00:12:13 INFO 00:12:13 INFO [loop_until]: kubectl --namespace=xlou top node 00:12:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:12:13 INFO [loop_until]: OK (rc = 0) 00:12:13 DEBUG --- stdout --- 00:12:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5996Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 806m 2% 6633Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 48m 0% 2590Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 1484m 4% 10354Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 617m 2% 8672Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 135m 0% 5181Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 52m 0% 2590Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 8591m 28% 9153Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1403m 4% 9325Mi 8% gke-xlou-cdm-ds-0f601751-j488 479m 1% 6040Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 1241m 4% 9733Mi 8% gke-xlou-cdm-ds-0f601751-nj6q 936m 3% 6521Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 806m 2% 7910Mi 7% gke-xlou-cdm-frontend-a8771548-i82n 3034m 19% 4212Mi 7% 00:12:13 DEBUG --- stderr --- 00:12:13 DEBUG 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "7.1008168049056914" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG 00:13:13 DEBUG --- status code --- 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481993" 00:13:13 DEBUG 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "1000.5666666666666" 00:13:13 DEBUG ] 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:13:13 DEBUG } 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG ] 00:13:13 DEBUG { 00:13:13 DEBUG } 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG } 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "1000.6333333333333" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "ds-idrepo-0" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "ds-idrepo-2" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "ds-idrepo-0" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "ds-idrepo-2" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "instance": "10.142.0.107:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "instance": "10.142.0.43:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0.024333333333333054" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG { 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "1000.9" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 ERROR [http_cmd]: ERROR 00:13:13 ERROR --- status code --- 00:13:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:13:13 ERROR --- http response --- 00:13:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:13:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "2781638656" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "ds-idrepo-0" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "ds-cts-2" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "118317241.94016221" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "ds-cts-0" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "ds-cts-2" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "385965.5" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481993" 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "instance": "10.142.0.107:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "instance": "10.142.0.43:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "28744131.666666664" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "ds-cts-0" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "ds-cts-2" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "ds-idrepo-0" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:13:13 DEBUG 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG "pod": "ds-idrepo-2" 00:13:13 DEBUG { 00:13:13 DEBUG }, 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "value": [ 00:13:13 DEBUG "data": { 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "0" 00:13:13 DEBUG "result": [ 00:13:13 DEBUG ] 00:13:13 DEBUG { 00:13:13 DEBUG } 00:13:13 DEBUG "metric": { 00:13:13 DEBUG ] 00:13:13 DEBUG "pod": "ds-cts-0" 00:13:13 DEBUG } 00:13:13 DEBUG }, 00:13:13 DEBUG } 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "ds-cts-2" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0.00002766" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713481993" 00:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713481993" 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "instance": "10.142.0.107:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "instance": "10.142.0.43:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "0.03116666666666769" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "instance": "10.142.0.107:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "instance": "10.142.0.43:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "265307272192" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "16.000659788405336" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO [http_cmd]: http status code OK 00:13:13 DEBUG --- status code --- 00:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:13:13 DEBUG --- http response --- 00:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:13:13 DEBUG ----- output ----- 00:13:13 DEBUG { 00:13:13 DEBUG "status": "success", 00:13:13 DEBUG "data": { 00:13:13 DEBUG "resultType": "vector", 00:13:13 DEBUG "result": [ 00:13:13 DEBUG { 00:13:13 DEBUG "metric": { 00:13:13 DEBUG "instance": "10.142.0.107:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG [...] 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG 00:13:13 DEBUG "instance": "10.142.0.43:9100" 00:13:13 DEBUG }, 00:13:13 DEBUG "value": [ 00:13:13 DEBUG 1713481993, 00:13:13 DEBUG "28742297.733333334" 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG ] 00:13:13 DEBUG } 00:13:13 DEBUG } 00:13:13 INFO 00:13:13 INFO [loop_until]: kubectl --namespace=xlou top pods 00:13:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:13:13 INFO [loop_until]: OK (rc = 0) 00:13:13 DEBUG --- stdout --- 00:13:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 5498m 7158Mi am-776d56b7dd-m7qb7 4483m 7240Mi am-776d56b7dd-xr8wp 6484m 7404Mi ds-cts-0 3119m 7074Mi ds-cts-1 2945m 6962Mi ds-cts-2 2980m 7050Mi ds-idrepo-0 2722m 3250Mi ds-idrepo-1 2583m 2909Mi ds-idrepo-2 2520m 2986Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1168Mi idm-b85f9d978-wnhxb 4m 1214Mi lodemon-7dccb59994-hvp6g 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7998m 4065Mi 00:13:13 DEBUG --- stderr --- 00:13:13 DEBUG 00:13:13 INFO 00:13:13 INFO [loop_until]: kubectl --namespace=xlou top node 00:13:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:13:13 INFO [loop_until]: OK (rc = 0) 00:13:13 DEBUG --- stdout --- 00:13:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6011Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2559m 8% 6639Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 47m 0% 2600Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 5466m 18% 10349Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 2993m 10% 9369Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 5168Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2587Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 23195m 77% 11344Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 4748m 15% 9356Mi 8% gke-xlou-cdm-ds-0f601751-j488 2734m 9% 6023Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 3107m 10% 10660Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 2837m 9% 6526Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2796m 9% 8862Mi 7% gke-xlou-cdm-frontend-a8771548-i82n 8173m 51% 7313Mi 12% 00:13:13 DEBUG --- stderr --- 00:13:13 DEBUG 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:14:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:14:13 DEBUG }, 00:14:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:14:13 DEBUG 00:14:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:14:13 DEBUG }, 00:14:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "1010.9333333333333" 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "1011.0666666666666" 00:14:13 DEBUG ] 00:14:13 DEBUG "1010.8" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482053" 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "8.02666343291622" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:14:13 DEBUG "result": [ 00:14:13 DEBUG "result": [ 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG { 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:14:13 DEBUG "pod": "ds-idrepo-0" 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG }, 00:14:13 DEBUG }, 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "value": [ 00:14:13 DEBUG "value": [ 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "instance": "10.142.0.107:9100" 00:14:13 DEBUG [...] 00:14:13 DEBUG [...] 00:14:13 DEBUG "pod": "ds-idrepo-0" 00:14:13 DEBUG }, 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:14:13 DEBUG "pod": "ds-idrepo-2" 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG }, 00:14:13 DEBUG }, 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG "value": [ 00:14:13 DEBUG "value": [ 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "0" 00:14:13 DEBUG "0" 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG ] 00:14:13 DEBUG ] 00:14:13 DEBUG 00:14:13 DEBUG "instance": "10.142.0.43:9100" 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 DEBUG "pod": "ds-idrepo-2" 00:14:13 DEBUG }, 00:14:13 DEBUG ] 00:14:13 DEBUG ] 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 DEBUG "value": [ 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG "0.0190000000000003" 00:14:13 DEBUG "0" 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG ] 00:14:13 DEBUG ] 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 DEBUG { 00:14:13 DEBUG } 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "6089891840" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 ERROR [http_cmd]: ERROR 00:14:13 ERROR --- status code --- 00:14:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:14:13 ERROR --- http response --- 00:14:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:14:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "445887.5" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "ds-idrepo-0" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "ds-cts-2" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "ds-cts-0" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "ds-cts-2" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "instance": "10.142.0.107:9100" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "instance": "10.142.0.43:9100" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0.02746666666666423" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "instance": "10.142.0.107:9100" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "instance": "10.142.0.43:9100" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "27842256.233333334" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "ds-cts-0" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "ds-cts-2" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "ds-idrepo-0" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "ds-idrepo-2" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "ds-cts-0" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "ds-cts-2" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0.00002766" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482053" 00:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482053" 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "instance": "10.142.0.107:9100" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "instance": "10.142.0.43:9100" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "266557660160" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "pod": "lodemon-7dccb59994-hvp6g" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "0" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO [http_cmd]: http status code OK 00:14:13 DEBUG --- status code --- 00:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:14:13 DEBUG --- http response --- 00:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:14:13 DEBUG ----- output ----- 00:14:13 DEBUG { 00:14:13 DEBUG "status": "success", 00:14:13 DEBUG "data": { 00:14:13 DEBUG "resultType": "vector", 00:14:13 DEBUG "result": [ 00:14:13 DEBUG { 00:14:13 DEBUG "metric": { 00:14:13 DEBUG "instance": "10.142.0.107:9100" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG [...] 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG 00:14:13 DEBUG "instance": "10.142.0.43:9100" 00:14:13 DEBUG }, 00:14:13 DEBUG "value": [ 00:14:13 DEBUG 1713482053, 00:14:13 DEBUG "27842049.166666664" 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG ] 00:14:13 DEBUG } 00:14:13 DEBUG } 00:14:13 INFO 00:14:13 INFO [loop_until]: kubectl --namespace=xlou top pods 00:14:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:14:13 INFO 00:14:13 INFO [loop_until]: kubectl --namespace=xlou top node 00:14:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:14:13 INFO [loop_until]: OK (rc = 0) 00:14:13 DEBUG --- stdout --- 00:14:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 5381m 7160Mi am-776d56b7dd-m7qb7 4403m 7241Mi am-776d56b7dd-xr8wp 6619m 8754Mi ds-cts-0 3055m 7872Mi ds-cts-1 3095m 7774Mi ds-cts-2 3005m 8149Mi ds-idrepo-0 2684m 3250Mi ds-idrepo-1 2431m 2910Mi ds-idrepo-2 2510m 2986Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1168Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-7dccb59994-hvp6g 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7867m 6066Mi 00:14:13 DEBUG --- stderr --- 00:14:13 DEBUG 00:14:13 INFO [loop_until]: OK (rc = 0) 00:14:13 DEBUG --- stdout --- 00:14:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6013Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2620m 8% 6633Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 49m 0% 2593Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 5589m 18% 10354Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 3009m 10% 10344Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5190Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 54m 0% 2586Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 23126m 77% 12704Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 4801m 16% 9349Mi 8% gke-xlou-cdm-ds-0f601751-j488 2687m 9% 6040Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 3185m 10% 11684Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 2826m 9% 6520Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2999m 10% 9954Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 8207m 51% 9487Mi 16% 00:14:13 DEBUG --- stderr --- 00:14:13 DEBUG 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "8.024946280857478" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713482113" 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "ds-idrepo-0" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "ds-idrepo-2" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "1024.9666666666667" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "1024.9666666666667" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "1024.8666666666666" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713482113" 00:15:13 ERROR [http_cmd]: ERROR 00:15:13 ERROR --- status code --- 00:15:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:15:13 ERROR --- http response --- 00:15:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:15:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG }, 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG "value": [ 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG "7559786496" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "instance": "10.142.0.107:9100" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "instance": "10.142.0.43:9100" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0.02233333333333321" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "ds-idrepo-0" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "ds-idrepo-2" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "506171.5" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "ds-idrepo-0" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "ds-cts-2" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "ds-cts-0" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "ds-cts-2" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "instance": "10.142.0.107:9100" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "instance": "10.142.0.43:9100" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0.030666666666668143" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "ds-idrepo-0" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "ds-idrepo-2" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "instance": "10.142.0.107:9100" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "instance": "10.142.0.43:9100" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "28832577.933333334" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "instance": "10.142.0.107:9100" 00:15:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:15:13 DEBUG }, 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "instance": "10.142.0.43:9100" 00:15:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:15:13 DEBUG }, 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "267816340480" 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "ds-cts-0" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "ds-cts-2" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "pod": "ds-cts-0" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "pod": "ds-cts-2" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "0" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482113" 00:15:13 INFO [http_cmd]: http status code OK 00:15:13 DEBUG --- status code --- 00:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:15:13 DEBUG --- http response --- 00:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:15:13 DEBUG ----- output ----- 00:15:13 DEBUG { 00:15:13 DEBUG "status": "success", 00:15:13 DEBUG "data": { 00:15:13 DEBUG "resultType": "vector", 00:15:13 DEBUG "result": [ 00:15:13 DEBUG { 00:15:13 DEBUG "metric": { 00:15:13 DEBUG "instance": "10.142.0.107:9100" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG [...] 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG 00:15:13 DEBUG "instance": "10.142.0.43:9100" 00:15:13 DEBUG }, 00:15:13 DEBUG "value": [ 00:15:13 DEBUG 1713482113, 00:15:13 DEBUG "28831102.6" 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG ] 00:15:13 DEBUG } 00:15:13 DEBUG } 00:15:13 INFO 00:15:13 INFO [loop_until]: kubectl --namespace=xlou top pods 00:15:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:15:13 INFO [loop_until]: OK (rc = 0) 00:15:13 DEBUG --- stdout --- 00:15:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 5730m 7160Mi am-776d56b7dd-m7qb7 4452m 7239Mi am-776d56b7dd-xr8wp 6295m 8762Mi ds-cts-0 3099m 8537Mi ds-cts-1 3056m 8676Mi ds-cts-2 2928m 8805Mi ds-idrepo-0 2740m 3249Mi ds-idrepo-1 2649m 2910Mi ds-idrepo-2 2537m 2986Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1168Mi idm-b85f9d978-wnhxb 4m 1214Mi lodemon-7dccb59994-hvp6g 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7982m 7505Mi 00:15:13 DEBUG --- stderr --- 00:15:13 DEBUG 00:15:13 INFO 00:15:13 INFO [loop_until]: kubectl --namespace=xlou top node 00:15:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:15:14 INFO [loop_until]: OK (rc = 0) 00:15:14 DEBUG --- stdout --- 00:15:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6001Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2648m 8% 6640Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 50m 0% 2590Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 5767m 19% 10353Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 3168m 10% 10942Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 135m 0% 5165Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 54m 0% 2602Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 23222m 77% 12697Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 4850m 16% 9352Mi 8% gke-xlou-cdm-ds-0f601751-j488 2811m 9% 6023Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 3136m 10% 12437Mi 11% gke-xlou-cdm-ds-0f601751-nj6q 2866m 9% 6529Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2958m 9% 10664Mi 9% gke-xlou-cdm-frontend-a8771548-i82n 8217m 51% 10486Mi 17% 00:15:14 DEBUG --- stderr --- 00:15:14 DEBUG 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING No values available for selected timeframe. 00:16:11 WARNING File /tmp/lodemon_data-20240419_000409/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [19/Apr/2024 00:16:11] "GET /monitoring/average?start_time=20240418_231315&stop_time=20240418_231410 HTTP/1.1" 200 - 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "333.1333333333333" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "8.025123149732659" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "ds-idrepo-0" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "ds-idrepo-2" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "333.26666666666665" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482173" 00:16:13 ERROR [http_cmd]: ERROR 00:16:13 ERROR --- status code --- 00:16:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:16:13 ERROR --- http response --- 00:16:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:16:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "333.1333333333333" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "8393850880" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "instance": "10.142.0.107:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "instance": "10.142.0.43:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0.00799999999999983" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "ds-idrepo-0" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "ds-idrepo-2" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "8651087.71851619" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:16:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:16:13 DEBUG }, 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:16:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:16:13 DEBUG }, 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "556008" 00:16:13 DEBUG "0" 00:16:13 DEBUG ] 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "381.4727430771141" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "ds-idrepo-0" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "ds-cts-2" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "ds-cts-0" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "ds-cts-2" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "1375972.1842791508" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "instance": "10.142.0.107:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "instance": "10.142.0.43:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0.010366666666665006" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "instance": "10.142.0.107:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "instance": "10.142.0.43:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "268819223552" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "ds-idrepo-0" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "ds-idrepo-2" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "instance": "10.142.0.107:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "instance": "10.142.0.43:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "7625300.766666667" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "10.981888182381123" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "ds-cts-0" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "ds-cts-2" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "pod": "ds-cts-0" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "pod": "ds-cts-2" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "0" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482173" 00:16:13 INFO [http_cmd]: http status code OK 00:16:13 DEBUG --- status code --- 00:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:16:13 DEBUG --- http response --- 00:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:16:13 DEBUG ----- output ----- 00:16:13 DEBUG { 00:16:13 DEBUG "status": "success", 00:16:13 DEBUG "data": { 00:16:13 DEBUG "resultType": "vector", 00:16:13 DEBUG "result": [ 00:16:13 DEBUG { 00:16:13 DEBUG "metric": { 00:16:13 DEBUG "instance": "10.142.0.107:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG [...] 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG 00:16:13 DEBUG "instance": "10.142.0.43:9100" 00:16:13 DEBUG }, 00:16:13 DEBUG "value": [ 00:16:13 DEBUG 1713482173, 00:16:13 DEBUG "7622730.666666666" 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG ] 00:16:13 DEBUG } 00:16:13 DEBUG } 00:16:13 INFO 00:16:13 INFO [loop_until]: kubectl --namespace=xlou top pods 00:16:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:16:13 INFO [loop_until]: OK (rc = 0) 00:16:13 DEBUG --- stdout --- 00:16:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 4332m 8810Mi am-776d56b7dd-m7qb7 4094m 7241Mi am-776d56b7dd-xr8wp 4403m 8740Mi ds-cts-0 2389m 9683Mi ds-cts-1 3093m 9513Mi ds-cts-2 2753m 9665Mi ds-idrepo-0 2327m 3249Mi ds-idrepo-1 1841m 2909Mi ds-idrepo-2 2044m 2986Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1168Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-7dccb59994-hvp6g 7m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8239m 8137Mi 00:16:13 DEBUG --- stderr --- 00:16:13 DEBUG 00:16:14 INFO 00:16:14 INFO [loop_until]: kubectl --namespace=xlou top node 00:16:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:16:14 INFO [loop_until]: OK (rc = 0) 00:16:14 DEBUG --- stdout --- 00:16:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 5993Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2112m 7% 6637Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 47m 0% 2597Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 4411m 14% 11994Mi 10% gke-xlou-cdm-default-pool-1b0769c8-w15q 2685m 8% 12101Mi 10% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 5154Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2592Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 15097m 50% 11958Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 3701m 12% 9354Mi 8% gke-xlou-cdm-ds-0f601751-j488 2451m 8% 6011Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 2154m 7% 13187Mi 11% gke-xlou-cdm-ds-0f601751-nj6q 2199m 7% 6520Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2375m 7% 11525Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 7936m 49% 3958Mi 6% 00:16:14 DEBUG --- stderr --- 00:16:14 DEBUG 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0.03333333333333333" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0.06666666666666667" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 ERROR [http_cmd]: ERROR 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 ERROR --- status code --- 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 ERROR --- http response --- 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:17:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:17:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:17:13 DEBUG { 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG { 00:17:13 DEBUG "data": { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "result": [ 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG { 00:17:13 DEBUG "result": [ 00:17:13 DEBUG "metric": { 00:17:13 DEBUG { 00:17:13 DEBUG "pod": "ds-idrepo-0" 00:17:13 DEBUG "metric": { 00:17:13 DEBUG }, 00:17:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:17:13 DEBUG "value": [ 00:17:13 DEBUG }, 00:17:13 DEBUG 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "ds-idrepo-2" 00:17:13 DEBUG 00:17:13 DEBUG }, 00:17:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:17:13 DEBUG "value": [ 00:17:13 DEBUG }, 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG "0" 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG ] 00:17:13 DEBUG "0.06666666666666667" 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0.8000806193182736" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG } 00:17:13 DEBUG --- status code --- 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:17:13 DEBUG { 00:17:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:17:13 DEBUG ----- output ----- 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713482233" 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG { 00:17:13 DEBUG "data": { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "resultType": "vector", 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 DEBUG "data": { 00:17:13 DEBUG "result": [ 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG { 00:17:13 DEBUG "result": [ 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG "metric": { 00:17:13 DEBUG { 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG "instance": "10.142.0.107:9100" 00:17:13 DEBUG "metric": { 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG }, 00:17:13 DEBUG "pod": "ds-idrepo-0" 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG "value": [ 00:17:13 DEBUG }, 00:17:13 DEBUG 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG { 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "data": { 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "result": [ 00:17:13 DEBUG 00:17:13 DEBUG "instance": "10.142.0.43:9100" 00:17:13 DEBUG { 00:17:13 DEBUG 00:17:13 DEBUG }, 00:17:13 DEBUG "metric": { 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG "pod": "ds-idrepo-2" 00:17:13 DEBUG "value": [ 00:17:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG }, 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG }, 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG "value": [ 00:17:13 DEBUG "0" 00:17:13 DEBUG "value": [ 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG ] 00:17:13 DEBUG 00:17:13 DEBUG "0" 00:17:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:17:13 DEBUG } 00:17:13 DEBUG 00:17:13 DEBUG ] 00:17:13 DEBUG ] 00:17:13 DEBUG 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 DEBUG [...] 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG 00:17:13 DEBUG } 00:17:13 DEBUG 00:17:13 DEBUG } 00:17:13 DEBUG 00:17:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "111796224" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "25675201.285803847" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "561007.5" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "am-776d56b7dd-6jxzf" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "am-776d56b7dd-m7qb7" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "7001.642402149311" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "ds-idrepo-0" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "ds-cts-2" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "ds-cts-0" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "ds-cts-2" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "8382599.671933226" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "instance": "10.142.0.107:9100" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "instance": "10.142.0.43:9100" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "268945431552" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "instance": "10.142.0.107:9100" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "instance": "10.142.0.43:9100" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0.0006666666666679551" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "ds-idrepo-0" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "ds-idrepo-2" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "instance": "10.142.0.107:9100" 00:17:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:17:13 DEBUG }, 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "instance": "10.142.0.43:9100" 00:17:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:17:13 DEBUG }, 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "25560.466666666664" 00:17:13 DEBUG "1.2560734158762128" 00:17:13 DEBUG ] 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG { 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "ds-cts-0" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG [...] 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG 00:17:13 DEBUG "pod": "ds-cts-2" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "0" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713482233" 00:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713482233" 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG --- http response --- 00:17:13 INFO [http_cmd]: http status code OK 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:17:13 DEBUG --- status code --- 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:17:13 DEBUG { 00:17:13 DEBUG --- http response --- 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG "data": { 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG "result": [ 00:17:13 DEBUG { 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "ds-cts-0" 00:17:13 DEBUG }, 00:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:17:13 DEBUG "value": [ 00:17:13 DEBUG ----- output ----- 00:17:13 DEBUG 00:17:13 DEBUG { 00:17:13 DEBUG 00:17:13 DEBUG "status": "success", 00:17:13 DEBUG 00:17:13 DEBUG "data": { 00:17:13 DEBUG [...] 00:17:13 DEBUG "resultType": "vector", 00:17:13 DEBUG 00:17:13 DEBUG "result": [ 00:17:13 DEBUG 00:17:13 DEBUG { 00:17:13 DEBUG 00:17:13 DEBUG "metric": { 00:17:13 DEBUG "pod": "ds-cts-2" 00:17:13 DEBUG "instance": "10.142.0.107:9100" 00:17:13 DEBUG }, 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG 00:17:13 DEBUG "0" 00:17:13 DEBUG 00:17:13 DEBUG ] 00:17:13 DEBUG 00:17:13 DEBUG } 00:17:13 DEBUG [...] 00:17:13 DEBUG ] 00:17:13 DEBUG 00:17:13 DEBUG } 00:17:13 DEBUG 00:17:13 DEBUG } 00:17:13 DEBUG 00:17:13 DEBUG "instance": "10.142.0.43:9100" 00:17:13 DEBUG }, 00:17:13 DEBUG "value": [ 00:17:13 DEBUG 1713482233, 00:17:13 DEBUG "22319.300000000003" 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG ] 00:17:13 DEBUG } 00:17:13 DEBUG } 00:17:13 INFO 00:17:13 INFO [loop_until]: kubectl --namespace=xlou top pods 00:17:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:17:13 INFO [loop_until]: OK (rc = 0) 00:17:13 DEBUG --- stdout --- 00:17:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-776d56b7dd-6jxzf 6m 8808Mi am-776d56b7dd-m7qb7 18m 7223Mi am-776d56b7dd-xr8wp 7m 8722Mi ds-cts-0 105m 9680Mi ds-cts-1 104m 9512Mi ds-cts-2 44m 9725Mi ds-idrepo-0 229m 3262Mi ds-idrepo-1 142m 2957Mi ds-idrepo-2 247m 3015Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 10m 1169Mi idm-b85f9d978-wnhxb 6m 1215Mi lodemon-7dccb59994-hvp6g 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1m 15Mi 00:17:13 DEBUG --- stderr --- 00:17:13 DEBUG 00:17:14 INFO 00:17:14 INFO [loop_until]: kubectl --namespace=xlou top node 00:17:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:17:14 INFO [loop_until]: OK (rc = 0) 00:17:14 DEBUG --- stdout --- 00:17:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 6000Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 282m 0% 6608Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 51m 0% 2598Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 123m 0% 12013Mi 10% gke-xlou-cdm-default-pool-1b0769c8-w15q 167m 0% 12081Mi 10% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 5188Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2612Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 113m 0% 11871Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 83m 0% 9325Mi 8% gke-xlou-cdm-ds-0f601751-j488 76m 0% 6022Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 193m 0% 13167Mi 11% gke-xlou-cdm-ds-0f601751-nj6q 309m 1% 6490Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 141m 0% 11520Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 126m 0% 3923Mi 6% 00:17:14 DEBUG --- stderr --- 00:17:14 DEBUG 00:17:54 INFO Finished: True 00:17:54 INFO Waiting for threads to register finish flag 00:18:14 INFO Done. Have a nice day! :) 127.0.0.1 - - [19/Apr/2024 00:18:14] "GET /monitoring/stop HTTP/1.1" 200 - 00:18:16 WARNING File /tmp/lodemon_data-20240419_000409/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p5353', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6451', ...) 127.0.0.1 - - [19/Apr/2024 00:18:20] "GET /monitoring/process HTTP/1.1" 200 -