==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-5498c574fd-96grz Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-frontend-18a44ad7-wxs9/10.142.0.11 Start Time: Fri, 27 Dec 2024 18:34:42 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=5498c574fd skaffold.dev/run-id=45b042d1-5ad9-474a-ab79-99adfc6c6129 Annotations: Status: Running IP: 10.106.40.189 IPs: IP: 10.106.40.189 Controlled By: ReplicaSet/lodemon-5498c574fd Containers: lodemon: Container ID: containerd://d303abbb28ba0075a29235adb9cd71581198be9d4c169f56cbb28c4c45f376c7 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:4ba4f6066d128cbe60604a46e6c908ecfbc82c97c0cb359b8c9fdd92c9ab55a1 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Fri, 27 Dec 2024 18:34:43 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: idc ds-only Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9djhp (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-9djhp: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 04:48:29 INFO 04:48:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:48:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:48:30 INFO [loop_until]: OK (rc = 0) 04:48:30 DEBUG --- stdout --- 04:48:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 103m 13370Mi rcs-687d95c5b-qs2xv 115m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:48:30 DEBUG --- stderr --- 04:48:30 DEBUG 04:48:31 INFO 04:48:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:48:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:48:31 INFO [loop_until]: OK (rc = 0) 04:48:31 DEBUG --- stdout --- 04:48:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 205m 1% 3464Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 190m 0% 18868Mi 66% 04:48:31 DEBUG --- stderr --- 04:48:31 DEBUG 04:49:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361340" 04:49:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361340" 04:49:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361340" 04:49:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361340" 04:49:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361340" 04:49:00 INFO [http_cmd]: http status code OK 04:49:00 DEBUG --- status code --- 04:49:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:00 DEBUG --- http response --- 04:49:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:00 INFO [http_cmd]: http status code OK 04:49:00 INFO [http_cmd]: http status code OK 04:49:00 DEBUG --- status code --- 04:49:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:00 DEBUG --- http response --- 04:49:00 INFO [http_cmd]: http status code OK 04:49:00 INFO [http_cmd]: http status code OK 04:49:00 DEBUG --- status code --- 04:49:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:00 DEBUG --- status code --- 04:49:00 DEBUG --- status code --- 04:49:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:00 DEBUG --- http response --- 04:49:00 DEBUG --- http response --- 04:49:00 DEBUG --- http response --- 04:49:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361340" 04:49:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361340" 04:49:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361340" 04:49:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361340" 04:49:00 INFO [http_cmd]: http status code OK 04:49:00 DEBUG --- status code --- 04:49:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:00 INFO [http_cmd]: http status code OK 04:49:00 DEBUG --- http response --- 04:49:00 DEBUG --- status code --- 04:49:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:00 DEBUG --- http response --- 04:49:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:00 INFO [http_cmd]: http status code OK 04:49:00 DEBUG --- status code --- 04:49:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:00 DEBUG --- http response --- 04:49:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:00 INFO [http_cmd]: http status code OK 04:49:00 DEBUG --- status code --- 04:49:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:00 DEBUG --- http response --- 04:49:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361341" 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 ERROR [http_cmd]: ERROR 04:49:01 ERROR --- status code --- 04:49:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:49:01 ERROR --- http response --- 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361341" 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG { 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG "data": { 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG "result": [ 04:49:01 DEBUG { 04:49:01 DEBUG "metric": { 04:49:01 DEBUG "pod": "ds-idrepo-0" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG [...] 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG "830959616" 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG { 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG "data": { 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG "result": [ 04:49:01 DEBUG { 04:49:01 DEBUG "metric": { 04:49:01 DEBUG "instance": "10.142.0.11:9100" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG [...] 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG "instance": "10.142.0.51:9100" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG "32085.333333333336" 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361341" 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG { 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361341" 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG "data": { 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG "result": [ 04:49:01 DEBUG { 04:49:01 DEBUG "metric": { 04:49:01 DEBUG "instance": "10.142.0.11:9100" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG [...] 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG "instance": "10.142.0.51:9100" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG "0.0004999999999995454" 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG { 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG "data": { 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG "result": [ 04:49:01 DEBUG { 04:49:01 DEBUG "metric": { 04:49:01 DEBUG "instance": "10.142.0.11:9100" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG [...] 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG "instance": "10.142.0.51:9100" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG "2164915.133333334" 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG { 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:49:01 DEBUG "data": { 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG { 04:49:01 DEBUG "result": [ 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG { 04:49:01 DEBUG "data": { 04:49:01 DEBUG "metric": { 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG "instance": "10.142.0.11:9100" 04:49:01 DEBUG "result": [ 04:49:01 DEBUG }, 04:49:01 DEBUG { 04:49:01 DEBUG "value": [ 04:49:01 DEBUG "metric": { 04:49:01 DEBUG 04:49:01 DEBUG "pod": "ds-idrepo-0" 04:49:01 DEBUG 04:49:01 DEBUG }, 04:49:01 DEBUG 04:49:01 DEBUG "value": [ 04:49:01 DEBUG [...] 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG [...] 04:49:01 DEBUG "instance": "10.142.0.51:9100" 04:49:01 DEBUG 04:49:01 DEBUG }, 04:49:01 DEBUG 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:49:01 DEBUG "2183776.3000000003" 04:49:01 DEBUG }, 04:49:01 DEBUG ] 04:49:01 DEBUG "value": [ 04:49:01 DEBUG } 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG ] 04:49:01 DEBUG "992.4654230440838" 04:49:01 DEBUG } 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361341" 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG { 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG "data": { 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG "result": [ 04:49:01 DEBUG { 04:49:01 DEBUG "metric": { 04:49:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG [...] 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG "14875299840" 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:49:01 DEBUG ] 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG } 04:49:01 DEBUG { 04:49:01 DEBUG } 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG "data": { 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG "result": [ 04:49:01 DEBUG { 04:49:01 DEBUG "metric": { 04:49:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG [...] 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:49:01 DEBUG }, 04:49:01 DEBUG "value": [ 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG "0.0971831644567741" 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361341" 04:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361341" 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG { 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG "data": { 04:49:01 DEBUG "resultType": "vector", 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG "result": [ 04:49:01 DEBUG { 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG "metric": { 04:49:01 INFO [http_cmd]: http status code OK 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG "pod": "ds-idrepo-0" 04:49:01 DEBUG --- status code --- 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG }, 04:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:49:01 DEBUG "value": [ 04:49:01 DEBUG --- http response --- 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG 04:49:01 DEBUG { 04:49:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:49:01 DEBUG 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG ----- output ----- 04:49:01 DEBUG 04:49:01 DEBUG "data": { 04:49:01 DEBUG { 04:49:01 DEBUG [...] 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG "status": "success", 04:49:01 DEBUG 04:49:01 DEBUG "result": [ 04:49:01 DEBUG "data": { 04:49:01 DEBUG 04:49:01 DEBUG { 04:49:01 DEBUG "resultType": "vector", 04:49:01 DEBUG 04:49:01 DEBUG "metric": { 04:49:01 DEBUG "result": [ 04:49:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:49:01 DEBUG "instance": "10.142.0.11:9100" 04:49:01 DEBUG { 04:49:01 DEBUG }, 04:49:01 DEBUG }, 04:49:01 DEBUG "metric": { 04:49:01 DEBUG "value": [ 04:49:01 DEBUG "value": [ 04:49:01 DEBUG "pod": "ds-idrepo-0" 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG 04:49:01 DEBUG }, 04:49:01 DEBUG "0.001194088746340032" 04:49:01 DEBUG 04:49:01 DEBUG "value": [ 04:49:01 DEBUG ] 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG } 04:49:01 DEBUG [...] 04:49:01 DEBUG 04:49:01 DEBUG ] 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG } 04:49:01 DEBUG 04:49:01 DEBUG [...] 04:49:01 DEBUG } 04:49:01 DEBUG 04:49:01 DEBUG 04:49:01 DEBUG "instance": "10.142.0.51:9100" 04:49:01 DEBUG 04:49:01 DEBUG }, 04:49:01 DEBUG 04:49:01 DEBUG "value": [ 04:49:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG }, 04:49:01 DEBUG "0.00033333333333303023" 04:49:01 DEBUG "value": [ 04:49:01 DEBUG ] 04:49:01 DEBUG 1735361341, 04:49:01 DEBUG } 04:49:01 DEBUG "0" 04:49:01 DEBUG ] 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:01 DEBUG ] 04:49:01 DEBUG } 04:49:01 DEBUG } 04:49:30 INFO 04:49:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:49:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:49:30 INFO [loop_until]: OK (rc = 0) 04:49:30 DEBUG --- stdout --- 04:49:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 89m 13369Mi rcs-687d95c5b-qs2xv 63m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:49:30 DEBUG --- stderr --- 04:49:30 DEBUG 04:49:31 INFO 04:49:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:49:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:49:31 INFO [loop_until]: OK (rc = 0) 04:49:31 DEBUG --- stdout --- 04:49:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 221m 1% 3429Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 183m 0% 18863Mi 66% 04:49:31 DEBUG --- stderr --- 04:49:31 DEBUG 04:50:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361400" 04:50:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361400" 04:50:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361400" 04:50:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361400" 04:50:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361400" 04:50:00 INFO [http_cmd]: http status code OK 04:50:00 DEBUG --- status code --- 04:50:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:00 DEBUG --- http response --- 04:50:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:00 INFO [http_cmd]: http status code OK 04:50:00 DEBUG --- status code --- 04:50:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:00 DEBUG --- http response --- 04:50:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:00 INFO [http_cmd]: http status code OK 04:50:00 INFO [http_cmd]: http status code OK 04:50:00 INFO [http_cmd]: http status code OK 04:50:00 DEBUG --- status code --- 04:50:00 DEBUG --- status code --- 04:50:00 DEBUG --- status code --- 04:50:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:00 DEBUG --- http response --- 04:50:00 DEBUG --- http response --- 04:50:00 DEBUG --- http response --- 04:50:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361400" 04:50:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361400" 04:50:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361400" 04:50:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361400" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 ERROR [http_cmd]: ERROR 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 ERROR --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 ERROR --- http response --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361401" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG { 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "data": { 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "result": [ 04:50:01 DEBUG { 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "pod": "ds-idrepo-0" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG [...] 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG "830959616" 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG { 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "data": { 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "result": [ 04:50:01 DEBUG { 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "instance": "10.142.0.11:9100" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG [...] 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG "instance": "10.142.0.51:9100" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG "19933.86666666667" 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361401" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG { 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "data": { 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "result": [ 04:50:01 DEBUG { 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "instance": "10.142.0.11:9100" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG [...] 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG "instance": "10.142.0.51:9100" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG "0.0003999999999990679" 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG { 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "data": { 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "result": [ 04:50:01 DEBUG { 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "instance": "10.142.0.11:9100" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG [...] 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG "instance": "10.142.0.51:9100" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG "1216728.8333333333" 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG { 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "data": { 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "result": [ 04:50:01 DEBUG { 04:50:01 DEBUG "metric": { 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG "instance": "10.142.0.11:9100" 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG }, 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG "value": [ 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:50:01 DEBUG 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG 04:50:01 DEBUG { 04:50:01 DEBUG [...] 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG 04:50:01 DEBUG "data": { 04:50:01 DEBUG 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG 04:50:01 DEBUG "result": [ 04:50:01 DEBUG "instance": "10.142.0.51:9100" 04:50:01 DEBUG { 04:50:01 DEBUG }, 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "value": [ 04:50:01 DEBUG "pod": "ds-idrepo-0" 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG }, 04:50:01 DEBUG "1237468.366666667" 04:50:01 DEBUG "value": [ 04:50:01 DEBUG ] 04:50:01 DEBUG 04:50:01 DEBUG } 04:50:01 DEBUG 04:50:01 DEBUG ] 04:50:01 DEBUG 04:50:01 DEBUG } 04:50:01 DEBUG [...] 04:50:01 DEBUG } 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG "919.9450954535979" 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361401" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG { 04:50:01 DEBUG { 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "data": { 04:50:01 DEBUG "data": { 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "result": [ 04:50:01 DEBUG "result": [ 04:50:01 DEBUG { 04:50:01 DEBUG { 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:50:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:50:01 DEBUG }, 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG [...] 04:50:01 DEBUG [...] 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:50:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:50:01 DEBUG }, 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG "14873739264" 04:50:01 DEBUG "0.16747260168877995" 04:50:01 DEBUG ] 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 DEBUG ] 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361401" 04:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361401" 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG { 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "data": { 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "result": [ 04:50:01 DEBUG { 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "pod": "ds-idrepo-0" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG [...] 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG "0.0009670023139147219" 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 INFO [http_cmd]: http status code OK 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG --- status code --- 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG --- http response --- 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:50:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG ----- output ----- 04:50:01 DEBUG { 04:50:01 DEBUG { 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "status": "success", 04:50:01 DEBUG "data": { 04:50:01 DEBUG "data": { 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "resultType": "vector", 04:50:01 DEBUG "result": [ 04:50:01 DEBUG "result": [ 04:50:01 DEBUG { 04:50:01 DEBUG { 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "metric": { 04:50:01 DEBUG "instance": "10.142.0.11:9100" 04:50:01 DEBUG "pod": "ds-idrepo-0" 04:50:01 DEBUG }, 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG [...] 04:50:01 DEBUG [...] 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG 04:50:01 DEBUG "instance": "10.142.0.51:9100" 04:50:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:50:01 DEBUG }, 04:50:01 DEBUG }, 04:50:01 DEBUG "value": [ 04:50:01 DEBUG "value": [ 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG 1735361401, 04:50:01 DEBUG "0.0003333333333335039" 04:50:01 DEBUG "0" 04:50:01 DEBUG ] 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 DEBUG ] 04:50:01 DEBUG ] 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:01 DEBUG } 04:50:30 INFO 04:50:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:50:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:50:30 INFO [loop_until]: OK (rc = 0) 04:50:30 DEBUG --- stdout --- 04:50:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 84m 13369Mi rcs-687d95c5b-qs2xv 70m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:50:30 DEBUG --- stderr --- 04:50:30 DEBUG 04:50:31 INFO 04:50:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:50:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:50:31 INFO [loop_until]: OK (rc = 0) 04:50:31 DEBUG --- stdout --- 04:50:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 182m 1% 3445Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 130m 0% 18863Mi 66% 04:50:31 DEBUG --- stderr --- 04:50:31 DEBUG 04:51:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361460" 04:51:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361460" 04:51:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361460" 04:51:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361460" 04:51:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361460" 04:51:00 INFO [http_cmd]: http status code OK 04:51:00 DEBUG --- status code --- 04:51:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:00 INFO [http_cmd]: http status code OK 04:51:00 DEBUG --- http response --- 04:51:00 DEBUG --- status code --- 04:51:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:00 DEBUG --- http response --- 04:51:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:00 INFO [http_cmd]: http status code OK 04:51:00 DEBUG --- status code --- 04:51:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:00 DEBUG --- http response --- 04:51:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:00 INFO [http_cmd]: http status code OK 04:51:00 DEBUG --- status code --- 04:51:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:00 DEBUG --- http response --- 04:51:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:00 INFO [http_cmd]: http status code OK 04:51:00 DEBUG --- status code --- 04:51:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:00 DEBUG --- http response --- 04:51:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 ERROR [http_cmd]: ERROR 04:51:01 ERROR --- status code --- 04:51:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:51:01 ERROR --- http response --- 04:51:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:51:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG { 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG "data": { 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG "result": [ 04:51:01 DEBUG { 04:51:01 DEBUG "metric": { 04:51:01 DEBUG "pod": "ds-idrepo-0" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG [...] 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG "830959616" 04:51:01 DEBUG ] 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG } 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG ] 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG } 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG } 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG { 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG "data": { 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG "result": [ 04:51:01 DEBUG { 04:51:01 DEBUG "metric": { 04:51:01 DEBUG "instance": "10.142.0.11:9100" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG [...] 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG "instance": "10.142.0.51:9100" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG "17885.86666666667" 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG { 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG "data": { 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG "result": [ 04:51:01 DEBUG { 04:51:01 DEBUG "metric": { 04:51:01 DEBUG "instance": "10.142.0.11:9100" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG [...] 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG "instance": "10.142.0.51:9100" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG "0.0011333333333321133" 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG { 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG "data": { 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG "result": [ 04:51:01 DEBUG { 04:51:01 DEBUG "metric": { 04:51:01 DEBUG "instance": "10.142.0.11:9100" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG [...] 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG "instance": "10.142.0.51:9100" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG "1240208.3333333335" 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG { 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG "data": { 04:51:01 DEBUG { 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG "result": [ 04:51:01 DEBUG "data": { 04:51:01 DEBUG { 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG "metric": { 04:51:01 DEBUG "result": [ 04:51:01 DEBUG "pod": "ds-idrepo-0" 04:51:01 DEBUG { 04:51:01 DEBUG }, 04:51:01 DEBUG "metric": { 04:51:01 DEBUG "value": [ 04:51:01 DEBUG "instance": "10.142.0.11:9100" 04:51:01 DEBUG 04:51:01 DEBUG }, 04:51:01 DEBUG 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG [...] 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG [...] 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:51:01 DEBUG 04:51:01 DEBUG }, 04:51:01 DEBUG 04:51:01 DEBUG "value": [ 04:51:01 DEBUG "instance": "10.142.0.51:9100" 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG }, 04:51:01 DEBUG "784.9657936661398" 04:51:01 DEBUG "value": [ 04:51:01 DEBUG ] 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG } 04:51:01 DEBUG "1257771.066666667" 04:51:01 DEBUG ] 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG { 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:51:01 DEBUG "data": { 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG { 04:51:01 DEBUG "result": [ 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG { 04:51:01 DEBUG "data": { 04:51:01 DEBUG "metric": { 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:51:01 DEBUG "result": [ 04:51:01 DEBUG }, 04:51:01 DEBUG { 04:51:01 DEBUG "value": [ 04:51:01 DEBUG "metric": { 04:51:01 DEBUG 04:51:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:51:01 DEBUG 04:51:01 DEBUG }, 04:51:01 DEBUG 04:51:01 DEBUG [...] 04:51:01 DEBUG 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:51:01 DEBUG 04:51:01 DEBUG }, 04:51:01 DEBUG [...] 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG 04:51:01 DEBUG "14876377088" 04:51:01 DEBUG 04:51:01 DEBUG ] 04:51:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:51:01 DEBUG } 04:51:01 DEBUG }, 04:51:01 DEBUG ] 04:51:01 DEBUG "value": [ 04:51:01 DEBUG } 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG } 04:51:01 DEBUG "0.07228628413795739" 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361461" 04:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361461" 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG { 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG "data": { 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG "result": [ 04:51:01 DEBUG { 04:51:01 DEBUG "metric": { 04:51:01 DEBUG "pod": "ds-idrepo-0" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG [...] 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:51:01 DEBUG }, 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG "0.0011503432600117464" 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG --- http response --- 04:51:01 INFO [http_cmd]: http status code OK 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:51:01 DEBUG --- status code --- 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:01 DEBUG { 04:51:01 DEBUG --- http response --- 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:51:01 DEBUG "data": { 04:51:01 DEBUG ----- output ----- 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG { 04:51:01 DEBUG "result": [ 04:51:01 DEBUG "status": "success", 04:51:01 DEBUG { 04:51:01 DEBUG "data": { 04:51:01 DEBUG "metric": { 04:51:01 DEBUG "resultType": "vector", 04:51:01 DEBUG "instance": "10.142.0.11:9100" 04:51:01 DEBUG "result": [ 04:51:01 DEBUG }, 04:51:01 DEBUG { 04:51:01 DEBUG "value": [ 04:51:01 DEBUG "metric": { 04:51:01 DEBUG 04:51:01 DEBUG "pod": "ds-idrepo-0" 04:51:01 DEBUG 04:51:01 DEBUG }, 04:51:01 DEBUG 04:51:01 DEBUG "value": [ 04:51:01 DEBUG [...] 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG 04:51:01 DEBUG [...] 04:51:01 DEBUG "instance": "10.142.0.51:9100" 04:51:01 DEBUG 04:51:01 DEBUG }, 04:51:01 DEBUG 04:51:01 DEBUG "value": [ 04:51:01 DEBUG 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:51:01 DEBUG "0.0009999999999995644" 04:51:01 DEBUG }, 04:51:01 DEBUG ] 04:51:01 DEBUG "value": [ 04:51:01 DEBUG } 04:51:01 DEBUG 1735361461, 04:51:01 DEBUG ] 04:51:01 DEBUG "0" 04:51:01 DEBUG } 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:01 DEBUG ] 04:51:01 DEBUG } 04:51:01 DEBUG } 04:51:30 INFO 04:51:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:51:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:51:30 INFO [loop_until]: OK (rc = 0) 04:51:30 DEBUG --- stdout --- 04:51:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 71m 13370Mi rcs-687d95c5b-qs2xv 65m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:51:30 DEBUG --- stderr --- 04:51:30 DEBUG 04:51:31 INFO 04:51:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:51:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:51:32 INFO [loop_until]: OK (rc = 0) 04:51:32 DEBUG --- stdout --- 04:51:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 185m 1% 3456Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 143m 0% 18863Mi 66% 04:51:32 DEBUG --- stderr --- 04:51:32 DEBUG 04:52:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361520" 04:52:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361520" 04:52:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361520" 04:52:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361520" 04:52:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361520" 04:52:00 INFO [http_cmd]: http status code OK 04:52:00 DEBUG --- status code --- 04:52:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:00 DEBUG --- http response --- 04:52:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:00 INFO [http_cmd]: http status code OK 04:52:00 DEBUG --- status code --- 04:52:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:00 DEBUG --- http response --- 04:52:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:00 INFO [http_cmd]: http status code OK 04:52:00 DEBUG --- status code --- 04:52:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:00 DEBUG --- http response --- 04:52:00 INFO [http_cmd]: http status code OK 04:52:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:00 DEBUG --- status code --- 04:52:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:00 DEBUG --- http response --- 04:52:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:00 INFO [http_cmd]: http status code OK 04:52:00 DEBUG --- status code --- 04:52:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:00 DEBUG --- http response --- 04:52:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 DEBUG --- status code --- 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- http response --- 04:52:01 ERROR [http_cmd]: ERROR 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 ERROR --- status code --- 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 ERROR --- http response --- 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG --- status code --- 04:52:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:52:01 DEBUG { 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG { 04:52:01 DEBUG "data": { 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "data": { 04:52:01 DEBUG "result": [ 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG { 04:52:01 DEBUG "result": [ 04:52:01 DEBUG "metric": { 04:52:01 DEBUG { 04:52:01 DEBUG "pod": "ds-idrepo-0" 04:52:01 DEBUG "metric": { 04:52:01 DEBUG }, 04:52:01 DEBUG "instance": "10.142.0.11:9100" 04:52:01 DEBUG "value": [ 04:52:01 DEBUG }, 04:52:01 DEBUG 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG [...] 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG [...] 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:52:01 DEBUG 04:52:01 DEBUG }, 04:52:01 DEBUG "instance": "10.142.0.51:9100" 04:52:01 DEBUG "value": [ 04:52:01 DEBUG }, 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG "830959616" 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG ] 04:52:01 DEBUG "20070.400000000005" 04:52:01 DEBUG } 04:52:01 DEBUG ] 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG { 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "data": { 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "result": [ 04:52:01 DEBUG { 04:52:01 DEBUG "metric": { 04:52:01 DEBUG "instance": "10.142.0.11:9100" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG [...] 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG "instance": "10.142.0.51:9100" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG "0.0012999999999967337" 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG { 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "data": { 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "result": [ 04:52:01 DEBUG { 04:52:01 DEBUG "metric": { 04:52:01 DEBUG "instance": "10.142.0.11:9100" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG [...] 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG "instance": "10.142.0.51:9100" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG "1232931.866666667" 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG { 04:52:01 DEBUG { 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "data": { 04:52:01 DEBUG "data": { 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "result": [ 04:52:01 DEBUG "result": [ 04:52:01 DEBUG { 04:52:01 DEBUG { 04:52:01 DEBUG "metric": { 04:52:01 DEBUG "metric": { 04:52:01 DEBUG "instance": "10.142.0.11:9100" 04:52:01 DEBUG "pod": "ds-idrepo-0" 04:52:01 DEBUG }, 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG [...] 04:52:01 DEBUG [...] 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG "instance": "10.142.0.51:9100" 04:52:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:52:01 DEBUG }, 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG "1252509.1" 04:52:01 DEBUG "1058.8295919061836" 04:52:01 DEBUG ] 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 DEBUG ] 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG { 04:52:01 DEBUG { 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "data": { 04:52:01 DEBUG "data": { 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "result": [ 04:52:01 DEBUG "result": [ 04:52:01 DEBUG { 04:52:01 DEBUG { 04:52:01 DEBUG "metric": { 04:52:01 DEBUG "metric": { 04:52:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:52:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:52:01 DEBUG }, 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG [...] 04:52:01 DEBUG [...] 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:52:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:52:01 DEBUG }, 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG "0.12790872133966183" 04:52:01 DEBUG "14872944640" 04:52:01 DEBUG ] 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 DEBUG ] 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361521" 04:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361521" 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG { 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "data": { 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "result": [ 04:52:01 DEBUG { 04:52:01 DEBUG "metric": { 04:52:01 DEBUG "pod": "ds-idrepo-0" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG [...] 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG "0.0011634053274932551" 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG { 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "data": { 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "result": [ 04:52:01 DEBUG { 04:52:01 DEBUG "metric": { 04:52:01 DEBUG "instance": "10.142.0.11:9100" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG [...] 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG "instance": "10.142.0.51:9100" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG "0.0010000000000000382" 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:01 INFO [http_cmd]: http status code OK 04:52:01 DEBUG --- status code --- 04:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:52:01 DEBUG --- http response --- 04:52:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:52:01 DEBUG ----- output ----- 04:52:01 DEBUG { 04:52:01 DEBUG "status": "success", 04:52:01 DEBUG "data": { 04:52:01 DEBUG "resultType": "vector", 04:52:01 DEBUG "result": [ 04:52:01 DEBUG { 04:52:01 DEBUG "metric": { 04:52:01 DEBUG "pod": "ds-idrepo-0" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG [...] 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG 04:52:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:52:01 DEBUG }, 04:52:01 DEBUG "value": [ 04:52:01 DEBUG 1735361521, 04:52:01 DEBUG "0" 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG ] 04:52:01 DEBUG } 04:52:01 DEBUG } 04:52:30 INFO 04:52:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:52:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:52:30 INFO [loop_until]: OK (rc = 0) 04:52:30 DEBUG --- stdout --- 04:52:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 78m 13370Mi rcs-687d95c5b-qs2xv 62m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:52:30 DEBUG --- stderr --- 04:52:30 DEBUG 04:52:32 INFO 04:52:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:52:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:52:32 INFO [loop_until]: OK (rc = 0) 04:52:32 DEBUG --- stdout --- 04:52:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 185m 1% 3452Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 148m 0% 18859Mi 66% 04:52:32 DEBUG --- stderr --- 04:52:32 DEBUG 04:53:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361580" 04:53:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361580" 04:53:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361580" 04:53:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361580" 04:53:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361580" 04:53:00 INFO [http_cmd]: http status code OK 04:53:00 DEBUG --- status code --- 04:53:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:00 INFO [http_cmd]: http status code OK 04:53:00 DEBUG --- http response --- 04:53:00 DEBUG --- status code --- 04:53:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:00 DEBUG --- http response --- 04:53:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:00 INFO [http_cmd]: http status code OK 04:53:00 DEBUG --- status code --- 04:53:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:00 DEBUG --- http response --- 04:53:00 INFO [http_cmd]: http status code OK 04:53:00 INFO [http_cmd]: http status code OK 04:53:00 DEBUG --- status code --- 04:53:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:00 DEBUG --- status code --- 04:53:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:00 DEBUG --- http response --- 04:53:00 DEBUG --- http response --- 04:53:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 ERROR [http_cmd]: ERROR 04:53:01 ERROR --- status code --- 04:53:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:53:01 ERROR --- http response --- 04:53:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:53:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG { 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "data": { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "result": [ 04:53:01 DEBUG { 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "instance": "10.142.0.11:9100" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG "instance": "10.142.0.51:9100" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG --- status code --- 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG "18841.600000000006" 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG ] 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG } 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:53:01 DEBUG ] 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG } 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 DEBUG { 04:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:53:01 DEBUG } 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "data": { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "result": [ 04:53:01 DEBUG { 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "pod": "ds-idrepo-0" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG "830959616" 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361581" 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG { 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "data": { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "result": [ 04:53:01 DEBUG { 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "instance": "10.142.0.11:9100" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361581" 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG "instance": "10.142.0.51:9100" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG "0.00036666666666936487" 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG { 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "data": { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "result": [ 04:53:01 DEBUG { 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "instance": "10.142.0.11:9100" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG "instance": "10.142.0.51:9100" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG "1301356.0999999999" 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG { 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG "data": { 04:53:01 DEBUG { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "result": [ 04:53:01 DEBUG "data": { 04:53:01 DEBUG { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "result": [ 04:53:01 DEBUG "pod": "ds-idrepo-0" 04:53:01 DEBUG { 04:53:01 DEBUG }, 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "value": [ 04:53:01 DEBUG "instance": "10.142.0.11:9100" 04:53:01 DEBUG 04:53:01 DEBUG }, 04:53:01 DEBUG 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:53:01 DEBUG 04:53:01 DEBUG }, 04:53:01 DEBUG 04:53:01 DEBUG "value": [ 04:53:01 DEBUG "instance": "10.142.0.51:9100" 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG }, 04:53:01 DEBUG "713.4623143018445" 04:53:01 DEBUG "value": [ 04:53:01 DEBUG ] 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG } 04:53:01 DEBUG "1321043.466666667" 04:53:01 DEBUG ] 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361581" 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG { 04:53:01 DEBUG { 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "data": { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "result": [ 04:53:01 DEBUG "data": { 04:53:01 DEBUG { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:53:01 DEBUG "result": [ 04:53:01 DEBUG }, 04:53:01 DEBUG { 04:53:01 DEBUG "value": [ 04:53:01 DEBUG "metric": { 04:53:01 DEBUG 04:53:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:53:01 DEBUG 04:53:01 DEBUG }, 04:53:01 DEBUG 04:53:01 DEBUG "value": [ 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:53:01 DEBUG 04:53:01 DEBUG }, 04:53:01 DEBUG 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:53:01 DEBUG "14874165248" 04:53:01 DEBUG }, 04:53:01 DEBUG ] 04:53:01 DEBUG "value": [ 04:53:01 DEBUG } 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG ] 04:53:01 DEBUG "0.15174867830398528" 04:53:01 DEBUG } 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361581" 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361581" 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG { 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "data": { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "result": [ 04:53:01 DEBUG { 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "pod": "ds-idrepo-0" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG "0.0012505451796545307" 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG { 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "data": { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "result": [ 04:53:01 DEBUG { 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "instance": "10.142.0.11:9100" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG "instance": "10.142.0.51:9100" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG "0.0003333333333335039" 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:01 INFO [http_cmd]: http status code OK 04:53:01 DEBUG --- status code --- 04:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:53:01 DEBUG --- http response --- 04:53:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:53:01 DEBUG ----- output ----- 04:53:01 DEBUG { 04:53:01 DEBUG "status": "success", 04:53:01 DEBUG "data": { 04:53:01 DEBUG "resultType": "vector", 04:53:01 DEBUG "result": [ 04:53:01 DEBUG { 04:53:01 DEBUG "metric": { 04:53:01 DEBUG "pod": "ds-idrepo-0" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG [...] 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG 04:53:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:53:01 DEBUG }, 04:53:01 DEBUG "value": [ 04:53:01 DEBUG 1735361581, 04:53:01 DEBUG "0" 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG ] 04:53:01 DEBUG } 04:53:01 DEBUG } 04:53:30 INFO 04:53:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:53:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:53:30 INFO [loop_until]: OK (rc = 0) 04:53:30 DEBUG --- stdout --- 04:53:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 533Mi ds-idrepo-0 59m 13369Mi rcs-687d95c5b-qs2xv 83m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:53:30 DEBUG --- stderr --- 04:53:30 DEBUG 04:53:32 INFO 04:53:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:53:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:53:32 INFO [loop_until]: OK (rc = 0) 04:53:32 DEBUG --- stdout --- 04:53:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 205m 1% 3456Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 182m 0% 18863Mi 66% 04:53:32 DEBUG --- stderr --- 04:53:32 DEBUG 04:54:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361640" 04:54:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361640" 04:54:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361640" 04:54:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361640" 04:54:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361640" 04:54:00 INFO [http_cmd]: http status code OK 04:54:00 DEBUG --- status code --- 04:54:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:00 DEBUG --- http response --- 04:54:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:00 INFO [http_cmd]: http status code OK 04:54:00 DEBUG --- status code --- 04:54:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:00 INFO [http_cmd]: http status code OK 04:54:00 INFO [http_cmd]: http status code OK 04:54:00 DEBUG --- http response --- 04:54:00 INFO [http_cmd]: http status code OK 04:54:00 DEBUG --- status code --- 04:54:00 DEBUG --- status code --- 04:54:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:00 DEBUG --- status code --- 04:54:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:00 DEBUG --- http response --- 04:54:00 DEBUG --- http response --- 04:54:00 DEBUG --- http response --- 04:54:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG --- status code --- 04:54:01 ERROR [http_cmd]: ERROR 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 ERROR --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- http response --- 04:54:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 ERROR --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:54:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361641" 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- http response --- 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- http response --- 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:54:01 DEBUG { 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG { 04:54:01 DEBUG "data": { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "result": [ 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG { 04:54:01 DEBUG "result": [ 04:54:01 DEBUG "metric": { 04:54:01 DEBUG { 04:54:01 DEBUG "pod": "ds-idrepo-0" 04:54:01 DEBUG "metric": { 04:54:01 DEBUG }, 04:54:01 DEBUG "instance": "10.142.0.11:9100" 04:54:01 DEBUG "value": [ 04:54:01 DEBUG }, 04:54:01 DEBUG 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:54:01 DEBUG 04:54:01 DEBUG }, 04:54:01 DEBUG "instance": "10.142.0.51:9100" 04:54:01 DEBUG "value": [ 04:54:01 DEBUG }, 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG "830959616" 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG ] 04:54:01 DEBUG "26624.000000000004" 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361641" 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "result": [ 04:54:01 DEBUG { 04:54:01 DEBUG "metric": { 04:54:01 DEBUG "instance": "10.142.0.11:9100" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "instance": "10.142.0.51:9100" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "0.0005000000000033349" 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "result": [ 04:54:01 DEBUG { 04:54:01 DEBUG "metric": { 04:54:01 DEBUG "instance": "10.142.0.11:9100" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "instance": "10.142.0.51:9100" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "2316298.6000000006" 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361641" 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "result": [ 04:54:01 DEBUG { 04:54:01 DEBUG "metric": { 04:54:01 DEBUG "pod": "ds-idrepo-0" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "812.2044114925413" 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "result": [ 04:54:01 DEBUG { 04:54:01 DEBUG "metric": { 04:54:01 DEBUG "instance": "10.142.0.11:9100" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "instance": "10.142.0.51:9100" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "2335245.1333333342" 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361641" 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "result": [ 04:54:01 DEBUG { 04:54:01 DEBUG "metric": { 04:54:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "0.12982398152288444" 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "result": [ 04:54:01 DEBUG { 04:54:01 DEBUG "metric": { 04:54:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "14874017792" 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361641" 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361641" 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "result": [ 04:54:01 DEBUG { 04:54:01 DEBUG "metric": { 04:54:01 DEBUG "pod": "ds-idrepo-0" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "0.0012217398038619457" 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "result": [ 04:54:01 DEBUG { 04:54:01 DEBUG "metric": { 04:54:01 DEBUG "instance": "10.142.0.11:9100" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "instance": "10.142.0.51:9100" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "0.0003333333333335039" 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:01 INFO [http_cmd]: http status code OK 04:54:01 DEBUG --- status code --- 04:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:54:01 DEBUG --- http response --- 04:54:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:54:01 DEBUG ----- output ----- 04:54:01 DEBUG { 04:54:01 DEBUG "status": "success", 04:54:01 DEBUG "data": { 04:54:01 DEBUG "resultType": "vector", 04:54:01 DEBUG "result": [ 04:54:01 DEBUG { 04:54:01 DEBUG "metric": { 04:54:01 DEBUG "pod": "ds-idrepo-0" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG [...] 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG 04:54:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:54:01 DEBUG }, 04:54:01 DEBUG "value": [ 04:54:01 DEBUG 1735361641, 04:54:01 DEBUG "0" 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG ] 04:54:01 DEBUG } 04:54:01 DEBUG } 04:54:30 INFO 04:54:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:54:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:54:30 INFO [loop_until]: OK (rc = 0) 04:54:30 DEBUG --- stdout --- 04:54:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 579m 13372Mi rcs-687d95c5b-qs2xv 117m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:54:30 DEBUG --- stderr --- 04:54:30 DEBUG 04:54:32 INFO 04:54:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:54:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:54:32 INFO [loop_until]: OK (rc = 0) 04:54:32 DEBUG --- stdout --- 04:54:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 223m 1% 3449Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 651m 2% 18869Mi 66% 04:54:32 DEBUG --- stderr --- 04:54:32 DEBUG 04:55:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361700" 04:55:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361700" 04:55:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361700" 04:55:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361700" 04:55:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361700" 04:55:00 INFO [http_cmd]: http status code OK 04:55:00 INFO [http_cmd]: http status code OK 04:55:00 INFO [http_cmd]: http status code OK 04:55:00 INFO [http_cmd]: http status code OK 04:55:00 INFO [http_cmd]: http status code OK 04:55:00 DEBUG --- status code --- 04:55:00 DEBUG --- status code --- 04:55:00 DEBUG --- status code --- 04:55:00 DEBUG --- status code --- 04:55:00 DEBUG --- status code --- 04:55:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:00 DEBUG --- http response --- 04:55:00 DEBUG --- http response --- 04:55:00 DEBUG --- http response --- 04:55:00 DEBUG --- http response --- 04:55:00 DEBUG --- http response --- 04:55:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG --- status code --- 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- http response --- 04:55:01 ERROR [http_cmd]: ERROR 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 ERROR --- status code --- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 ERROR --- http response --- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- http response --- 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:55:01 DEBUG ----- output ----- 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG { 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG { 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG "data": { 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG "data": { 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG "result": [ 04:55:01 DEBUG "result": [ 04:55:01 DEBUG { 04:55:01 DEBUG { 04:55:01 DEBUG "metric": { 04:55:01 DEBUG "pod": "ds-idrepo-0" 04:55:01 DEBUG "metric": { 04:55:01 DEBUG }, 04:55:01 DEBUG "instance": "10.142.0.11:9100" 04:55:01 DEBUG "value": [ 04:55:01 DEBUG }, 04:55:01 DEBUG 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG [...] 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG [...] 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:55:01 DEBUG 04:55:01 DEBUG }, 04:55:01 DEBUG "instance": "10.142.0.51:9100" 04:55:01 DEBUG "value": [ 04:55:01 DEBUG }, 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG "830959616" 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG ] 04:55:01 DEBUG "24985.600000000006" 04:55:01 DEBUG } 04:55:01 DEBUG ] 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361701" 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG { 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG "data": { 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG "result": [ 04:55:01 DEBUG { 04:55:01 DEBUG "metric": { 04:55:01 DEBUG "instance": "10.142.0.11:9100" 04:55:01 DEBUG }, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG [...] 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG "instance": "10.142.0.51:9100" 04:55:01 DEBUG }, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG "0.0006000000000009702" 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG { 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG "data": { 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG "result": [ 04:55:01 DEBUG { 04:55:01 DEBUG "metric": { 04:55:01 DEBUG "instance": "10.142.0.11:9100" 04:55:01 DEBUG }, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG [...] 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG "instance": "10.142.0.51:9100" 04:55:01 DEBUG }, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG "1216366.5333333334" 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361701" 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG "data": { 04:55:01 DEBUG { 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG "result": [ 04:55:01 DEBUG "data": { 04:55:01 DEBUG { 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG "metric": { 04:55:01 DEBUG "result": [ 04:55:01 DEBUG "instance": "10.142.0.11:9100" 04:55:01 DEBUG { 04:55:01 DEBUG }, 04:55:01 DEBUG "metric": { 04:55:01 DEBUG "value": [ 04:55:01 DEBUG "pod": "ds-idrepo-0" 04:55:01 DEBUG 04:55:01 DEBUG }, 04:55:01 DEBUG 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG [...] 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG [...] 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG "instance": "10.142.0.51:9100" 04:55:01 DEBUG 04:55:01 DEBUG }, 04:55:01 DEBUG 04:55:01 DEBUG "value": [ 04:55:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG }, 04:55:01 DEBUG "1235350.1000000006" 04:55:01 DEBUG "value": [ 04:55:01 DEBUG ] 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG } 04:55:01 DEBUG "645.9814661013113" 04:55:01 DEBUG ] 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361701" 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG { 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG "data": { 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG "result": [ 04:55:01 DEBUG { 04:55:01 DEBUG "metric": { 04:55:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:55:01 DEBUG }, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG [...] 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG }, 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG "value": [ 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG "0.08630226713616324" 04:55:01 DEBUG { 04:55:01 DEBUG ] 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG } 04:55:01 DEBUG "data": { 04:55:01 DEBUG ] 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG } 04:55:01 DEBUG "result": [ 04:55:01 DEBUG } 04:55:01 DEBUG { 04:55:01 DEBUG "metric": { 04:55:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:55:01 DEBUG }, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG [...] 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:55:01 DEBUG }, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG "14876733440" 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361701" 04:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361701" 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG { 04:55:01 DEBUG "status": "success", 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG "data": { 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG "result": [ 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG { 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:55:01 DEBUG "metric": { 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG "instance": "10.142.0.11:9100" 04:55:01 DEBUG { 04:55:01 DEBUG }, 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG "value": [ 04:55:01 DEBUG "data": { 04:55:01 DEBUG 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG 04:55:01 DEBUG "result": [ 04:55:01 DEBUG 04:55:01 DEBUG { 04:55:01 DEBUG [...] 04:55:01 DEBUG "metric": { 04:55:01 DEBUG 04:55:01 DEBUG "pod": "ds-idrepo-0" 04:55:01 DEBUG 04:55:01 DEBUG }, 04:55:01 DEBUG 04:55:01 DEBUG "value": [ 04:55:01 DEBUG "instance": "10.142.0.51:9100" 04:55:01 DEBUG 04:55:01 DEBUG }, 04:55:01 DEBUG 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG [...] 04:55:01 DEBUG "0.00033333333333303023" 04:55:01 DEBUG 04:55:01 DEBUG ] 04:55:01 DEBUG 04:55:01 DEBUG } 04:55:01 DEBUG 04:55:01 DEBUG ] 04:55:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:55:01 DEBUG } 04:55:01 DEBUG }, 04:55:01 DEBUG } 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG "0.0012721005101293363" 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:01 INFO [http_cmd]: http status code OK 04:55:01 DEBUG --- status code --- 04:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:55:01 DEBUG --- http response --- 04:55:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:55:01 DEBUG ----- output ----- 04:55:01 DEBUG { 04:55:01 DEBUG "status": "success", 04:55:01 DEBUG "data": { 04:55:01 DEBUG "resultType": "vector", 04:55:01 DEBUG "result": [ 04:55:01 DEBUG { 04:55:01 DEBUG "metric": { 04:55:01 DEBUG "pod": "ds-idrepo-0" 04:55:01 DEBUG }, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG [...] 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG 04:55:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:55:01 DEBUG }, 04:55:01 DEBUG "value": [ 04:55:01 DEBUG 1735361701, 04:55:01 DEBUG "0" 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG ] 04:55:01 DEBUG } 04:55:01 DEBUG } 04:55:30 INFO 04:55:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:55:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:55:30 INFO [loop_until]: OK (rc = 0) 04:55:30 DEBUG --- stdout --- 04:55:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 66m 13372Mi rcs-687d95c5b-qs2xv 61m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:55:30 DEBUG --- stderr --- 04:55:30 DEBUG 04:55:32 INFO 04:55:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:55:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:55:32 INFO [loop_until]: OK (rc = 0) 04:55:32 DEBUG --- stdout --- 04:55:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 187m 1% 3420Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 139m 0% 18864Mi 66% 04:55:32 DEBUG --- stderr --- 04:55:32 DEBUG 04:56:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361760" 04:56:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361760" 04:56:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361760" 04:56:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361760" 04:56:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361760" 04:56:00 INFO [http_cmd]: http status code OK 04:56:00 DEBUG --- status code --- 04:56:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:00 DEBUG --- http response --- 04:56:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:00 INFO [http_cmd]: http status code OK 04:56:00 DEBUG --- status code --- 04:56:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:00 DEBUG --- http response --- 04:56:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:00 INFO [http_cmd]: http status code OK 04:56:00 DEBUG --- status code --- 04:56:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:00 DEBUG --- http response --- 04:56:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:00 INFO [http_cmd]: http status code OK 04:56:00 INFO [http_cmd]: http status code OK 04:56:00 DEBUG --- status code --- 04:56:00 DEBUG --- status code --- 04:56:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:00 DEBUG --- http response --- 04:56:00 DEBUG --- http response --- 04:56:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG --- status code --- 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 ERROR [http_cmd]: ERROR 04:56:01 ERROR --- status code --- 04:56:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:56:01 ERROR --- http response --- 04:56:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:56:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 DEBUG --- http response --- 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361761" 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG --- status code --- 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG --- status code --- 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:56:01 DEBUG { 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG { 04:56:01 DEBUG "data": { 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG "data": { 04:56:01 DEBUG "result": [ 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG { 04:56:01 DEBUG "result": [ 04:56:01 DEBUG "metric": { 04:56:01 DEBUG { 04:56:01 DEBUG "pod": "ds-idrepo-0" 04:56:01 DEBUG "metric": { 04:56:01 DEBUG }, 04:56:01 DEBUG "instance": "10.142.0.11:9100" 04:56:01 DEBUG "value": [ 04:56:01 DEBUG }, 04:56:01 DEBUG 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:56:01 DEBUG 04:56:01 DEBUG }, 04:56:01 DEBUG "instance": "10.142.0.51:9100" 04:56:01 DEBUG "value": [ 04:56:01 DEBUG }, 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG "value": [ 04:56:01 DEBUG "831000576" 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG ] 04:56:01 DEBUG "24029.86666666667" 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361761" 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG { 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG "data": { 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG "result": [ 04:56:01 DEBUG { 04:56:01 DEBUG "metric": { 04:56:01 DEBUG "instance": "10.142.0.11:9100" 04:56:01 DEBUG }, 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG "instance": "10.142.0.51:9100" 04:56:01 DEBUG }, 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG "0.0006333333333297258" 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG { 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG "data": { 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG "result": [ 04:56:01 DEBUG { 04:56:01 DEBUG "metric": { 04:56:01 DEBUG "instance": "10.142.0.11:9100" 04:56:01 DEBUG }, 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG "instance": "10.142.0.51:9100" 04:56:01 DEBUG }, 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG "1227973.9333333333" 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG { 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG "data": { 04:56:01 DEBUG { 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG "result": [ 04:56:01 DEBUG "data": { 04:56:01 DEBUG { 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG "metric": { 04:56:01 DEBUG "result": [ 04:56:01 DEBUG "instance": "10.142.0.11:9100" 04:56:01 DEBUG { 04:56:01 DEBUG }, 04:56:01 DEBUG "metric": { 04:56:01 DEBUG "value": [ 04:56:01 DEBUG "pod": "ds-idrepo-0" 04:56:01 DEBUG 04:56:01 DEBUG }, 04:56:01 DEBUG 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG "instance": "10.142.0.51:9100" 04:56:01 DEBUG 04:56:01 DEBUG }, 04:56:01 DEBUG 04:56:01 DEBUG "value": [ 04:56:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG }, 04:56:01 DEBUG "1245674.5000000002" 04:56:01 DEBUG "value": [ 04:56:01 DEBUG ] 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG } 04:56:01 DEBUG "987.0343303319142" 04:56:01 DEBUG ] 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361761" 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:56:01 DEBUG { 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG { 04:56:01 DEBUG "data": { 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG "data": { 04:56:01 DEBUG "result": [ 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG { 04:56:01 DEBUG "result": [ 04:56:01 DEBUG "metric": { 04:56:01 DEBUG { 04:56:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:56:01 DEBUG "metric": { 04:56:01 DEBUG }, 04:56:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:56:01 DEBUG "value": [ 04:56:01 DEBUG }, 04:56:01 DEBUG 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:56:01 DEBUG 04:56:01 DEBUG }, 04:56:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:56:01 DEBUG "value": [ 04:56:01 DEBUG }, 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG "value": [ 04:56:01 DEBUG "14874025984" 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG ] 04:56:01 DEBUG "0.08260566808257518" 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361761" 04:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361761" 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG { 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG "data": { 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG "result": [ 04:56:01 DEBUG { 04:56:01 DEBUG "metric": { 04:56:01 DEBUG "instance": "10.142.0.11:9100" 04:56:01 DEBUG }, 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG "instance": "10.142.0.51:9100" 04:56:01 DEBUG }, 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG "0" 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG --- http response --- 04:56:01 INFO [http_cmd]: http status code OK 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:56:01 DEBUG --- status code --- 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:56:01 DEBUG { 04:56:01 DEBUG --- http response --- 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:56:01 DEBUG "data": { 04:56:01 DEBUG ----- output ----- 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG { 04:56:01 DEBUG "result": [ 04:56:01 DEBUG "status": "success", 04:56:01 DEBUG { 04:56:01 DEBUG "data": { 04:56:01 DEBUG "metric": { 04:56:01 DEBUG "resultType": "vector", 04:56:01 DEBUG "pod": "ds-idrepo-0" 04:56:01 DEBUG "result": [ 04:56:01 DEBUG }, 04:56:01 DEBUG { 04:56:01 DEBUG "value": [ 04:56:01 DEBUG "metric": { 04:56:01 DEBUG 04:56:01 DEBUG "pod": "ds-idrepo-0" 04:56:01 DEBUG 04:56:01 DEBUG }, 04:56:01 DEBUG 04:56:01 DEBUG "value": [ 04:56:01 DEBUG [...] 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG 04:56:01 DEBUG [...] 04:56:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:56:01 DEBUG 04:56:01 DEBUG }, 04:56:01 DEBUG 04:56:01 DEBUG "value": [ 04:56:01 DEBUG 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:56:01 DEBUG "0" 04:56:01 DEBUG }, 04:56:01 DEBUG ] 04:56:01 DEBUG "value": [ 04:56:01 DEBUG } 04:56:01 DEBUG 1735361761, 04:56:01 DEBUG ] 04:56:01 DEBUG "0.0011917166437100707" 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:01 DEBUG ] 04:56:01 DEBUG } 04:56:01 DEBUG } 04:56:30 INFO 04:56:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:56:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:56:30 INFO [loop_until]: OK (rc = 0) 04:56:30 DEBUG --- stdout --- 04:56:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 45m 13372Mi rcs-687d95c5b-qs2xv 47m 425Mi rcs-687d95c5b-xxxmk 2m 395Mi 04:56:30 DEBUG --- stderr --- 04:56:30 DEBUG 04:56:32 INFO 04:56:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:56:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:56:32 INFO [loop_until]: OK (rc = 0) 04:56:32 DEBUG --- stdout --- 04:56:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 165m 1% 3418Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 103m 0% 18869Mi 66% 04:56:32 DEBUG --- stderr --- 04:56:32 DEBUG 04:57:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361820" 04:57:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361820" 04:57:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361820" 04:57:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361820" 04:57:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361820" 04:57:00 INFO [http_cmd]: http status code OK 04:57:00 INFO [http_cmd]: http status code OK 04:57:00 INFO [http_cmd]: http status code OK 04:57:00 DEBUG --- status code --- 04:57:00 INFO [http_cmd]: http status code OK 04:57:00 DEBUG --- status code --- 04:57:00 INFO [http_cmd]: http status code OK 04:57:00 DEBUG --- status code --- 04:57:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:00 DEBUG --- status code --- 04:57:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:00 DEBUG --- status code --- 04:57:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:00 DEBUG --- http response --- 04:57:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:00 DEBUG --- http response --- 04:57:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:00 DEBUG --- http response --- 04:57:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:00 DEBUG --- http response --- 04:57:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:00 DEBUG --- http response --- 04:57:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 ERROR [http_cmd]: ERROR 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 ERROR --- status code --- 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 ERROR --- http response --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG --- status code --- 04:57:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG --- http response --- 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361821" 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361821" 04:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG { 04:57:01 DEBUG { 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "data": { 04:57:01 DEBUG "data": { 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "result": [ 04:57:01 DEBUG "result": [ 04:57:01 DEBUG { 04:57:01 DEBUG { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "pod": "ds-idrepo-0" 04:57:01 DEBUG "instance": "10.142.0.11:9100" 04:57:01 DEBUG }, 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG [...] 04:57:01 DEBUG [...] 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:57:01 DEBUG "instance": "10.142.0.51:9100" 04:57:01 DEBUG }, 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG "831000576" 04:57:01 DEBUG "22664.533333333336" 04:57:01 DEBUG ] 04:57:01 DEBUG ] 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG ] 04:57:01 DEBUG ] 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG { 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "data": { 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "result": [ 04:57:01 DEBUG { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "instance": "10.142.0.11:9100" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG [...] 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG "instance": "10.142.0.51:9100" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG "0.0006333333333373049" 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG { 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "data": { 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "result": [ 04:57:01 DEBUG { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "instance": "10.142.0.11:9100" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG [...] 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG "instance": "10.142.0.51:9100" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG "2373784.933333334" 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG { 04:57:01 DEBUG { 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "data": { 04:57:01 DEBUG "data": { 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "result": [ 04:57:01 DEBUG "result": [ 04:57:01 DEBUG { 04:57:01 DEBUG { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "instance": "10.142.0.11:9100" 04:57:01 DEBUG "pod": "ds-idrepo-0" 04:57:01 DEBUG }, 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG [...] 04:57:01 DEBUG [...] 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG "instance": "10.142.0.51:9100" 04:57:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:57:01 DEBUG }, 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG "2393950.6000000006" 04:57:01 DEBUG "1080.391258062181" 04:57:01 DEBUG ] 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 DEBUG ] 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361821" 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG { 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "data": { 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "result": [ 04:57:01 DEBUG { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG [...] 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:57:01 DEBUG }, 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG "value": [ 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG "14873710592" 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG ] 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:57:01 DEBUG } 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG ] 04:57:01 DEBUG { 04:57:01 DEBUG } 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG } 04:57:01 DEBUG "data": { 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "result": [ 04:57:01 DEBUG { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG [...] 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG "0.03547943234339847" 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361821" 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG { 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "data": { 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "result": [ 04:57:01 DEBUG { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "instance": "10.142.0.11:9100" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG [...] 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG "instance": "10.142.0.51:9100" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG "0.0006666666666665342" 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG { 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "data": { 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "result": [ 04:57:01 DEBUG { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "pod": "ds-idrepo-0" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG [...] 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG "0" 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:01 INFO [http_cmd]: http status code OK 04:57:01 DEBUG --- status code --- 04:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:57:01 DEBUG --- http response --- 04:57:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:57:01 DEBUG ----- output ----- 04:57:01 DEBUG { 04:57:01 DEBUG "status": "success", 04:57:01 DEBUG "data": { 04:57:01 DEBUG "resultType": "vector", 04:57:01 DEBUG "result": [ 04:57:01 DEBUG { 04:57:01 DEBUG "metric": { 04:57:01 DEBUG "pod": "ds-idrepo-0" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG [...] 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG 04:57:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:57:01 DEBUG }, 04:57:01 DEBUG "value": [ 04:57:01 DEBUG 1735361821, 04:57:01 DEBUG "0.0012132320199664125" 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG ] 04:57:01 DEBUG } 04:57:01 DEBUG } 04:57:30 INFO 04:57:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:57:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:57:30 INFO [loop_until]: OK (rc = 0) 04:57:30 DEBUG --- stdout --- 04:57:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 148m 13373Mi rcs-687d95c5b-qs2xv 165m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:57:30 DEBUG --- stderr --- 04:57:30 DEBUG 04:57:32 INFO 04:57:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:57:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:57:32 INFO [loop_until]: OK (rc = 0) 04:57:32 DEBUG --- stdout --- 04:57:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 249m 1% 3422Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 210m 0% 18866Mi 66% 04:57:32 DEBUG --- stderr --- 04:57:32 DEBUG 04:58:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361880" 04:58:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361880" 04:58:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361880" 04:58:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361880" 04:58:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361880" 04:58:00 INFO [http_cmd]: http status code OK 04:58:00 DEBUG --- status code --- 04:58:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:00 DEBUG --- http response --- 04:58:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:00 INFO [http_cmd]: http status code OK 04:58:00 DEBUG --- status code --- 04:58:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:00 DEBUG --- http response --- 04:58:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:00 INFO [http_cmd]: http status code OK 04:58:00 DEBUG --- status code --- 04:58:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:00 DEBUG --- http response --- 04:58:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:00 INFO [http_cmd]: http status code OK 04:58:00 DEBUG --- status code --- 04:58:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:00 DEBUG --- http response --- 04:58:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:00 INFO [http_cmd]: http status code OK 04:58:00 DEBUG --- status code --- 04:58:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:00 DEBUG --- http response --- 04:58:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 ERROR [http_cmd]: ERROR 04:58:01 ERROR --- status code --- 04:58:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:58:01 ERROR --- http response --- 04:58:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:58:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG { 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG "data": { 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG "result": [ 04:58:01 DEBUG { 04:58:01 DEBUG "metric": { 04:58:01 DEBUG "instance": "10.142.0.11:9100" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG [...] 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:58:01 DEBUG "instance": "10.142.0.51:9100" 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG }, 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG { 04:58:01 DEBUG "value": [ 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG "data": { 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:58:01 DEBUG "23893.333333333336" 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG ] 04:58:01 DEBUG "result": [ 04:58:01 DEBUG { 04:58:01 DEBUG } 04:58:01 DEBUG { 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG ] 04:58:01 DEBUG "metric": { 04:58:01 DEBUG "data": { 04:58:01 DEBUG } 04:58:01 DEBUG "instance": "10.142.0.11:9100" 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG } 04:58:01 DEBUG }, 04:58:01 DEBUG "result": [ 04:58:01 DEBUG "value": [ 04:58:01 DEBUG { 04:58:01 DEBUG 04:58:01 DEBUG "metric": { 04:58:01 DEBUG 04:58:01 DEBUG "instance": "10.142.0.11:9100" 04:58:01 DEBUG 04:58:01 DEBUG }, 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG [...] 04:58:01 DEBUG "value": [ 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG "instance": "10.142.0.51:9100" 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:58:01 DEBUG [...] 04:58:01 DEBUG }, 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG 04:58:01 DEBUG "value": [ 04:58:01 DEBUG { 04:58:01 DEBUG 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG 04:58:01 DEBUG "0.00039999999999812045" 04:58:01 DEBUG "data": { 04:58:01 DEBUG "instance": "10.142.0.51:9100" 04:58:01 DEBUG ] 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG }, 04:58:01 DEBUG } 04:58:01 DEBUG "result": [ 04:58:01 DEBUG "value": [ 04:58:01 DEBUG ] 04:58:01 DEBUG { 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG } 04:58:01 DEBUG "metric": { 04:58:01 DEBUG "1459825.3" 04:58:01 DEBUG } 04:58:01 DEBUG "pod": "ds-idrepo-0" 04:58:01 DEBUG ] 04:58:01 DEBUG }, 04:58:01 DEBUG } 04:58:01 DEBUG "value": [ 04:58:01 DEBUG ] 04:58:01 DEBUG 04:58:01 DEBUG } 04:58:01 DEBUG 04:58:01 DEBUG } 04:58:01 DEBUG 04:58:01 DEBUG [...] 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG "831000576" 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG } 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG { 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG "data": { 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG "result": [ 04:58:01 DEBUG { 04:58:01 DEBUG "metric": { 04:58:01 DEBUG "instance": "10.142.0.11:9100" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG [...] 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG "instance": "10.142.0.51:9100" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG "1478611.2333333336" 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG { 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG "data": { 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG "result": [ 04:58:01 DEBUG { 04:58:01 DEBUG "metric": { 04:58:01 DEBUG "pod": "ds-idrepo-0" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG [...] 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG "948.2071124077302" 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG } 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG { 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG "data": { 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG "result": [ 04:58:01 DEBUG { 04:58:01 DEBUG "metric": { 04:58:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG [...] 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG "0.044110457145062565" 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG { 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG "data": { 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG "result": [ 04:58:01 DEBUG { 04:58:01 DEBUG "metric": { 04:58:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG [...] 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG "14875901952" 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG } 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361881" 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG { 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG "data": { 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG "result": [ 04:58:01 DEBUG { 04:58:01 DEBUG "metric": { 04:58:01 DEBUG "instance": "10.142.0.11:9100" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG [...] 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG "instance": "10.142.0.51:9100" 04:58:01 DEBUG }, 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG "0.0003333333333335039" 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG } 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 INFO [http_cmd]: http status code OK 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG --- status code --- 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG --- http response --- 04:58:01 DEBUG { 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG "data": { 04:58:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG ----- output ----- 04:58:01 DEBUG "result": [ 04:58:01 DEBUG { 04:58:01 DEBUG { 04:58:01 DEBUG "status": "success", 04:58:01 DEBUG "metric": { 04:58:01 DEBUG "data": { 04:58:01 DEBUG "pod": "ds-idrepo-0" 04:58:01 DEBUG "resultType": "vector", 04:58:01 DEBUG }, 04:58:01 DEBUG "result": [ 04:58:01 DEBUG "value": [ 04:58:01 DEBUG { 04:58:01 DEBUG 04:58:01 DEBUG "metric": { 04:58:01 DEBUG 04:58:01 DEBUG "pod": "ds-idrepo-0" 04:58:01 DEBUG 04:58:01 DEBUG }, 04:58:01 DEBUG [...] 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG 04:58:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:58:01 DEBUG [...] 04:58:01 DEBUG }, 04:58:01 DEBUG 04:58:01 DEBUG "value": [ 04:58:01 DEBUG 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG 04:58:01 DEBUG "0" 04:58:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:58:01 DEBUG ] 04:58:01 DEBUG }, 04:58:01 DEBUG } 04:58:01 DEBUG "value": [ 04:58:01 DEBUG ] 04:58:01 DEBUG 1735361881, 04:58:01 DEBUG } 04:58:01 DEBUG "0.0011396986249022783" 04:58:01 DEBUG } 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG ] 04:58:01 DEBUG } 04:58:01 DEBUG } 04:58:30 INFO 04:58:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:58:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:58:30 INFO [loop_until]: OK (rc = 0) 04:58:30 DEBUG --- stdout --- 04:58:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 98m 13372Mi rcs-687d95c5b-qs2xv 66m 425Mi rcs-687d95c5b-xxxmk 2m 395Mi 04:58:30 DEBUG --- stderr --- 04:58:30 DEBUG 04:58:32 INFO 04:58:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:58:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:58:32 INFO [loop_until]: OK (rc = 0) 04:58:32 DEBUG --- stdout --- 04:58:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 181m 1% 3448Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 134m 0% 18867Mi 66% 04:58:32 DEBUG --- stderr --- 04:58:32 DEBUG 04:59:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361940" 04:59:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361940" 04:59:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735361940" 04:59:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361940" 04:59:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361940" 04:59:00 INFO [http_cmd]: http status code OK 04:59:00 DEBUG --- status code --- 04:59:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:00 DEBUG --- http response --- 04:59:00 INFO [http_cmd]: http status code OK 04:59:00 DEBUG --- status code --- 04:59:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:00 DEBUG --- http response --- 04:59:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:00 INFO [http_cmd]: http status code OK 04:59:00 DEBUG --- status code --- 04:59:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:00 DEBUG --- http response --- 04:59:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:00 INFO [http_cmd]: http status code OK 04:59:00 DEBUG --- status code --- 04:59:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:00 DEBUG --- http response --- 04:59:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:00 INFO [http_cmd]: http status code OK 04:59:00 DEBUG --- status code --- 04:59:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:00 DEBUG --- http response --- 04:59:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735361941" 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 ERROR [http_cmd]: ERROR 04:59:01 ERROR --- status code --- 04:59:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:59:01 ERROR --- http response --- 04:59:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:59:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361941" 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361941" 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735361941" 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- status code --- 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG --- http response --- 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG { 04:59:01 DEBUG { 04:59:01 DEBUG { 04:59:01 DEBUG { 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "data": { 04:59:01 DEBUG "data": { 04:59:01 DEBUG "data": { 04:59:01 DEBUG "data": { 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "result": [ 04:59:01 DEBUG "result": [ 04:59:01 DEBUG "result": [ 04:59:01 DEBUG "result": [ 04:59:01 DEBUG { 04:59:01 DEBUG { 04:59:01 DEBUG { 04:59:01 DEBUG { 04:59:01 DEBUG "metric": { 04:59:01 DEBUG "metric": { 04:59:01 DEBUG "metric": { 04:59:01 DEBUG "metric": { 04:59:01 DEBUG "instance": "10.142.0.11:9100" 04:59:01 DEBUG "instance": "10.142.0.11:9100" 04:59:01 DEBUG "pod": "ds-idrepo-0" 04:59:01 DEBUG "instance": "10.142.0.11:9100" 04:59:01 DEBUG }, 04:59:01 DEBUG }, 04:59:01 DEBUG }, 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG "value": [ 04:59:01 DEBUG "value": [ 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG [...] 04:59:01 DEBUG [...] 04:59:01 DEBUG [...] 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG [...] 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:59:01 DEBUG "instance": "10.142.0.51:9100" 04:59:01 DEBUG "instance": "10.142.0.51:9100" 04:59:01 DEBUG 04:59:01 DEBUG }, 04:59:01 DEBUG }, 04:59:01 DEBUG }, 04:59:01 DEBUG "instance": "10.142.0.51:9100" 04:59:01 DEBUG "value": [ 04:59:01 DEBUG "value": [ 04:59:01 DEBUG "value": [ 04:59:01 DEBUG }, 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG "831000576" 04:59:01 DEBUG "26077.86666666667" 04:59:01 DEBUG "0.0006000000000000228" 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG ] 04:59:01 DEBUG ] 04:59:01 DEBUG ] 04:59:01 DEBUG "1230900.066666667" 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG ] 04:59:01 DEBUG ] 04:59:01 DEBUG ] 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:59:01 DEBUG { 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG { 04:59:01 DEBUG "data": { 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "data": { 04:59:01 DEBUG "result": [ 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG { 04:59:01 DEBUG "result": [ 04:59:01 DEBUG "metric": { 04:59:01 DEBUG { 04:59:01 DEBUG "instance": "10.142.0.11:9100" 04:59:01 DEBUG "metric": { 04:59:01 DEBUG }, 04:59:01 DEBUG "pod": "ds-idrepo-0" 04:59:01 DEBUG "value": [ 04:59:01 DEBUG }, 04:59:01 DEBUG 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG [...] 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG [...] 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG "instance": "10.142.0.51:9100" 04:59:01 DEBUG 04:59:01 DEBUG }, 04:59:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:59:01 DEBUG "value": [ 04:59:01 DEBUG }, 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG "1246514.5333333337" 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG ] 04:59:01 DEBUG "729.3196427849597" 04:59:01 DEBUG } 04:59:01 DEBUG ] 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735361941" 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG { 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "data": { 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "result": [ 04:59:01 DEBUG { 04:59:01 DEBUG "metric": { 04:59:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG [...] 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG "0.08983067178155145" 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG { 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "data": { 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "result": [ 04:59:01 DEBUG { 04:59:01 DEBUG "metric": { 04:59:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG [...] 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG "14875750400" 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735361941" 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG { 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "data": { 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "result": [ 04:59:01 DEBUG { 04:59:01 DEBUG "metric": { 04:59:01 DEBUG "instance": "10.142.0.11:9100" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG [...] 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG "instance": "10.142.0.51:9100" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG "0.0003333333333335039" 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG { 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "data": { 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "result": [ 04:59:01 DEBUG { 04:59:01 DEBUG "metric": { 04:59:01 DEBUG "pod": "ds-idrepo-0" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG [...] 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG "0" 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:01 INFO [http_cmd]: http status code OK 04:59:01 DEBUG --- status code --- 04:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:59:01 DEBUG --- http response --- 04:59:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 04:59:01 DEBUG ----- output ----- 04:59:01 DEBUG { 04:59:01 DEBUG "status": "success", 04:59:01 DEBUG "data": { 04:59:01 DEBUG "resultType": "vector", 04:59:01 DEBUG "result": [ 04:59:01 DEBUG { 04:59:01 DEBUG "metric": { 04:59:01 DEBUG "pod": "ds-idrepo-0" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG [...] 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG 04:59:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 04:59:01 DEBUG }, 04:59:01 DEBUG "value": [ 04:59:01 DEBUG 1735361941, 04:59:01 DEBUG "0.001168108224892698" 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG ] 04:59:01 DEBUG } 04:59:01 DEBUG } 04:59:30 INFO 04:59:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 04:59:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:59:30 INFO [loop_until]: OK (rc = 0) 04:59:30 DEBUG --- stdout --- 04:59:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 87m 13372Mi rcs-687d95c5b-qs2xv 1m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 04:59:30 DEBUG --- stderr --- 04:59:30 DEBUG 04:59:32 INFO 04:59:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 04:59:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:59:32 INFO [loop_until]: OK (rc = 0) 04:59:32 DEBUG --- stdout --- 04:59:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 163m 1% 3452Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 118m 0% 18864Mi 66% 04:59:32 DEBUG --- stderr --- 04:59:32 DEBUG 05:00:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362000" 05:00:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362000" 05:00:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362000" 05:00:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362000" 05:00:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362000" 05:00:00 INFO [http_cmd]: http status code OK 05:00:00 INFO [http_cmd]: http status code OK 05:00:00 INFO [http_cmd]: http status code OK 05:00:00 DEBUG --- status code --- 05:00:00 INFO [http_cmd]: http status code OK 05:00:00 DEBUG --- status code --- 05:00:00 INFO [http_cmd]: http status code OK 05:00:00 DEBUG --- status code --- 05:00:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:00 DEBUG --- status code --- 05:00:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:00 DEBUG --- status code --- 05:00:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:00 DEBUG --- http response --- 05:00:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:00 DEBUG --- http response --- 05:00:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:00 DEBUG --- http response --- 05:00:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:00 DEBUG --- http response --- 05:00:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:00 DEBUG --- http response --- 05:00:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362001" 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 ERROR [http_cmd]: ERROR 05:00:01 ERROR --- status code --- 05:00:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:00:01 ERROR --- http response --- 05:00:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:00:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362001" 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362001" 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362001" 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG { 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG "data": { 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "result": [ 05:00:01 DEBUG { 05:00:01 DEBUG "metric": { 05:00:01 DEBUG "instance": "10.142.0.11:9100" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG "instance": "10.142.0.51:9100" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG "32085.33333333334" 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG { 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG "data": { 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "result": [ 05:00:01 DEBUG { 05:00:01 DEBUG "metric": { 05:00:01 DEBUG "pod": "ds-idrepo-0" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG "831000576" 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG { 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG "data": { 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "result": [ 05:00:01 DEBUG { 05:00:01 DEBUG "metric": { 05:00:01 DEBUG "instance": "10.142.0.11:9100" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG "instance": "10.142.0.51:9100" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG "926855.2666666668" 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG { 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG "data": { 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "result": [ 05:00:01 DEBUG { 05:00:01 DEBUG "metric": { 05:00:01 DEBUG "instance": "10.142.0.11:9100" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG "instance": "10.142.0.51:9100" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG "0.0004999999999995453" 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG "data": { 05:00:01 DEBUG { 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG "result": [ 05:00:01 DEBUG "data": { 05:00:01 DEBUG { 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "metric": { 05:00:01 DEBUG "result": [ 05:00:01 DEBUG "instance": "10.142.0.11:9100" 05:00:01 DEBUG { 05:00:01 DEBUG }, 05:00:01 DEBUG "metric": { 05:00:01 DEBUG "value": [ 05:00:01 DEBUG "pod": "ds-idrepo-0" 05:00:01 DEBUG 05:00:01 DEBUG }, 05:00:01 DEBUG 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG "instance": "10.142.0.51:9100" 05:00:01 DEBUG 05:00:01 DEBUG }, 05:00:01 DEBUG 05:00:01 DEBUG "value": [ 05:00:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG }, 05:00:01 DEBUG "940137.3666666668" 05:00:01 DEBUG "value": [ 05:00:01 DEBUG ] 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG } 05:00:01 DEBUG "903.2091562846861" 05:00:01 DEBUG ] 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362001" 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG { 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG "data": { 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "result": [ 05:00:01 DEBUG { 05:00:01 DEBUG "metric": { 05:00:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG "0.0989002966208915" 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG { 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG "data": { 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "result": [ 05:00:01 DEBUG { 05:00:01 DEBUG "metric": { 05:00:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG "14873559040" 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362001" 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG { 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG "data": { 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "result": [ 05:00:01 DEBUG { 05:00:01 DEBUG "metric": { 05:00:01 DEBUG "instance": "10.142.0.11:9100" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG "instance": "10.142.0.51:9100" 05:00:01 DEBUG }, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG "0.0003333333333339776" 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 INFO [http_cmd]: http status code OK 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG --- status code --- 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG --- http response --- 05:00:01 DEBUG { 05:00:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:00:01 DEBUG ----- output ----- 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG { 05:00:01 DEBUG "data": { 05:00:01 DEBUG "status": "success", 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG "data": { 05:00:01 DEBUG "result": [ 05:00:01 DEBUG "resultType": "vector", 05:00:01 DEBUG { 05:00:01 DEBUG "result": [ 05:00:01 DEBUG "metric": { 05:00:01 DEBUG { 05:00:01 DEBUG "pod": "ds-idrepo-0" 05:00:01 DEBUG "metric": { 05:00:01 DEBUG }, 05:00:01 DEBUG "pod": "ds-idrepo-0" 05:00:01 DEBUG "value": [ 05:00:01 DEBUG }, 05:00:01 DEBUG 05:00:01 DEBUG "value": [ 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG [...] 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG 05:00:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:00:01 DEBUG 05:00:01 DEBUG }, 05:00:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:00:01 DEBUG "value": [ 05:00:01 DEBUG }, 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG "value": [ 05:00:01 DEBUG "0" 05:00:01 DEBUG 1735362001, 05:00:01 DEBUG ] 05:00:01 DEBUG "0.0011825528838094092" 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 DEBUG ] 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:01 DEBUG } 05:00:30 INFO 05:00:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:00:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:00:30 INFO [loop_until]: OK (rc = 0) 05:00:30 DEBUG --- stdout --- 05:00:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 47m 13372Mi rcs-687d95c5b-qs2xv 71m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:00:30 DEBUG --- stderr --- 05:00:30 DEBUG 05:00:32 INFO 05:00:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:00:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:00:32 INFO [loop_until]: OK (rc = 0) 05:00:32 DEBUG --- stdout --- 05:00:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 186m 1% 3472Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 139m 0% 18864Mi 66% 05:00:32 DEBUG --- stderr --- 05:00:32 DEBUG 05:01:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362060" 05:01:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362060" 05:01:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362060" 05:01:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362060" 05:01:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362060" 05:01:00 INFO [http_cmd]: http status code OK 05:01:00 DEBUG --- status code --- 05:01:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:00 DEBUG --- http response --- 05:01:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:00 INFO [http_cmd]: http status code OK 05:01:00 DEBUG --- status code --- 05:01:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:00 DEBUG --- http response --- 05:01:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:00 INFO [http_cmd]: http status code OK 05:01:00 INFO [http_cmd]: http status code OK 05:01:00 DEBUG --- status code --- 05:01:00 DEBUG --- status code --- 05:01:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:00 DEBUG --- http response --- 05:01:00 DEBUG --- http response --- 05:01:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:00 INFO [http_cmd]: http status code OK 05:01:00 DEBUG --- status code --- 05:01:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:00 DEBUG --- http response --- 05:01:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG --- status code --- 05:01:01 ERROR [http_cmd]: ERROR 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 ERROR --- status code --- 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG --- http response --- 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 ERROR --- http response --- 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- http response --- 05:01:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362061" 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362061" 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 DEBUG --- status code --- 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG --- http response --- 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG ----- output ----- 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG { 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG { 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG { 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG "data": { 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG "data": { 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG { 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:01:01 DEBUG "data": { 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG "result": [ 05:01:01 DEBUG "result": [ 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:01:01 DEBUG "data": { 05:01:01 DEBUG { 05:01:01 DEBUG { 05:01:01 DEBUG { 05:01:01 DEBUG "result": [ 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG "metric": { 05:01:01 DEBUG "metric": { 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG { 05:01:01 DEBUG { 05:01:01 DEBUG "result": [ 05:01:01 DEBUG "instance": "10.142.0.11:9100" 05:01:01 DEBUG "instance": "10.142.0.11:9100" 05:01:01 DEBUG "data": { 05:01:01 DEBUG "metric": { 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG { 05:01:01 DEBUG }, 05:01:01 DEBUG }, 05:01:01 DEBUG "value": [ 05:01:01 DEBUG "instance": "10.142.0.11:9100" 05:01:01 DEBUG "data": { 05:01:01 DEBUG "metric": { 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362061" 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG }, 05:01:01 DEBUG "value": [ 05:01:01 DEBUG "pod": "ds-idrepo-0" 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG 05:01:01 DEBUG }, 05:01:01 DEBUG "result": [ 05:01:01 DEBUG 05:01:01 DEBUG [...] 05:01:01 DEBUG "result": [ 05:01:01 DEBUG 05:01:01 DEBUG "value": [ 05:01:01 DEBUG { 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG { 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG "metric": { 05:01:01 DEBUG [...] 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG [...] 05:01:01 DEBUG 05:01:01 DEBUG "pod": "ds-idrepo-0" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362061" 05:01:01 DEBUG 05:01:01 DEBUG "metric": { 05:01:01 DEBUG "instance": "10.142.0.51:9100" 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG }, 05:01:01 DEBUG 05:01:01 DEBUG "instance": "10.142.0.11:9100" 05:01:01 DEBUG }, 05:01:01 DEBUG 05:01:01 DEBUG [...] 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 05:01:01 DEBUG }, 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG "instance": "10.142.0.51:9100" 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG "instance": "10.142.0.51:9100" 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG }, 05:01:01 DEBUG 05:01:01 DEBUG "181589.33333333337" 05:01:01 DEBUG }, 05:01:01 DEBUG 05:01:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 05:01:01 DEBUG ] 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG 05:01:01 DEBUG } 05:01:01 DEBUG }, 05:01:01 DEBUG "1253739.666666667" 05:01:01 DEBUG [...] 05:01:01 DEBUG "0.0009666666666703352" 05:01:01 DEBUG [...] 05:01:01 DEBUG ] 05:01:01 DEBUG "value": [ 05:01:01 DEBUG ] 05:01:01 DEBUG 05:01:01 DEBUG ] 05:01:01 DEBUG 05:01:01 DEBUG } 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG } 05:01:01 DEBUG 05:01:01 DEBUG } 05:01:01 DEBUG 05:01:01 DEBUG } 05:01:01 DEBUG "831000576" 05:01:01 DEBUG ] 05:01:01 DEBUG 05:01:01 DEBUG ] 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:01:01 DEBUG } 05:01:01 DEBUG 05:01:01 DEBUG ] 05:01:01 DEBUG } 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:01:01 DEBUG }, 05:01:01 DEBUG } 05:01:01 DEBUG "instance": "10.142.0.51:9100" 05:01:01 DEBUG } 05:01:01 DEBUG } 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG "value": [ 05:01:01 DEBUG }, 05:01:01 DEBUG ] 05:01:01 DEBUG { 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG "value": [ 05:01:01 DEBUG } 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG "809.4002161845572" 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG } 05:01:01 DEBUG "data": { 05:01:01 DEBUG ] 05:01:01 DEBUG "1239099.9333333333" 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG } 05:01:01 DEBUG ] 05:01:01 DEBUG "result": [ 05:01:01 DEBUG ] 05:01:01 DEBUG } 05:01:01 DEBUG } 05:01:01 DEBUG { 05:01:01 DEBUG } 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG "metric": { 05:01:01 DEBUG ] 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:01:01 DEBUG } 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG }, 05:01:01 DEBUG } 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG "value": [ 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:01:01 DEBUG 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG 05:01:01 DEBUG { 05:01:01 DEBUG 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG [...] 05:01:01 DEBUG "data": { 05:01:01 DEBUG 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG 05:01:01 DEBUG "result": [ 05:01:01 DEBUG 05:01:01 DEBUG { 05:01:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:01:01 DEBUG "metric": { 05:01:01 DEBUG }, 05:01:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:01:01 DEBUG "value": [ 05:01:01 DEBUG }, 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG "value": [ 05:01:01 DEBUG "0.12505678157565242" 05:01:01 DEBUG 05:01:01 DEBUG ] 05:01:01 DEBUG 05:01:01 DEBUG } 05:01:01 DEBUG 05:01:01 DEBUG ] 05:01:01 DEBUG [...] 05:01:01 DEBUG } 05:01:01 DEBUG 05:01:01 DEBUG } 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:01:01 DEBUG }, 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG "14874435584" 05:01:01 DEBUG ] 05:01:01 DEBUG } 05:01:01 DEBUG ] 05:01:01 DEBUG } 05:01:01 DEBUG } 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362061" 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG { 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG "data": { 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG "result": [ 05:01:01 DEBUG { 05:01:01 DEBUG "metric": { 05:01:01 DEBUG "instance": "10.142.0.11:9100" 05:01:01 DEBUG }, 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG [...] 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG "instance": "10.142.0.51:9100" 05:01:01 DEBUG }, 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG "0.0006666666666665342" 05:01:01 DEBUG ] 05:01:01 DEBUG } 05:01:01 DEBUG ] 05:01:01 DEBUG } 05:01:01 DEBUG } 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG { 05:01:01 INFO [http_cmd]: http status code OK 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG --- status code --- 05:01:01 DEBUG "data": { 05:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG --- http response --- 05:01:01 DEBUG "result": [ 05:01:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:01:01 DEBUG { 05:01:01 DEBUG ----- output ----- 05:01:01 DEBUG "metric": { 05:01:01 DEBUG { 05:01:01 DEBUG "pod": "ds-idrepo-0" 05:01:01 DEBUG "status": "success", 05:01:01 DEBUG }, 05:01:01 DEBUG "data": { 05:01:01 DEBUG "value": [ 05:01:01 DEBUG "resultType": "vector", 05:01:01 DEBUG 05:01:01 DEBUG "result": [ 05:01:01 DEBUG 05:01:01 DEBUG { 05:01:01 DEBUG 05:01:01 DEBUG "metric": { 05:01:01 DEBUG [...] 05:01:01 DEBUG "pod": "ds-idrepo-0" 05:01:01 DEBUG 05:01:01 DEBUG }, 05:01:01 DEBUG 05:01:01 DEBUG "value": [ 05:01:01 DEBUG 05:01:01 DEBUG 05:01:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:01:01 DEBUG 05:01:01 DEBUG }, 05:01:01 DEBUG 05:01:01 DEBUG "value": [ 05:01:01 DEBUG [...] 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG 05:01:01 DEBUG "0" 05:01:01 DEBUG 05:01:01 DEBUG ] 05:01:01 DEBUG 05:01:01 DEBUG } 05:01:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:01:01 DEBUG ] 05:01:01 DEBUG }, 05:01:01 DEBUG } 05:01:01 DEBUG "value": [ 05:01:01 DEBUG } 05:01:01 DEBUG 1735362061, 05:01:01 DEBUG "0.001183131721813275" 05:01:01 DEBUG ] 05:01:01 DEBUG } 05:01:01 DEBUG ] 05:01:01 DEBUG } 05:01:01 DEBUG } 05:01:30 INFO 05:01:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:01:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:01:30 INFO [loop_until]: OK (rc = 0) 05:01:30 DEBUG --- stdout --- 05:01:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 486m 13373Mi rcs-687d95c5b-qs2xv 122m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:01:30 DEBUG --- stderr --- 05:01:30 DEBUG 05:01:32 INFO 05:01:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:01:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:01:32 INFO [loop_until]: OK (rc = 0) 05:01:32 DEBUG --- stdout --- 05:01:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 205m 1% 3408Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 378m 1% 18865Mi 66% 05:01:32 DEBUG --- stderr --- 05:01:32 DEBUG 05:02:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362120" 05:02:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362120" 05:02:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362120" 05:02:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362120" 05:02:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362120" 05:02:00 INFO [http_cmd]: http status code OK 05:02:00 DEBUG --- status code --- 05:02:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:00 DEBUG --- http response --- 05:02:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:00 INFO [http_cmd]: http status code OK 05:02:00 INFO [http_cmd]: http status code OK 05:02:00 INFO [http_cmd]: http status code OK 05:02:00 DEBUG --- status code --- 05:02:00 INFO [http_cmd]: http status code OK 05:02:00 DEBUG --- status code --- 05:02:00 DEBUG --- status code --- 05:02:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:00 DEBUG --- status code --- 05:02:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:00 DEBUG --- http response --- 05:02:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:00 DEBUG --- http response --- 05:02:00 DEBUG --- http response --- 05:02:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:00 DEBUG --- http response --- 05:02:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG --- status code --- 05:02:01 ERROR [http_cmd]: ERROR 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 ERROR --- status code --- 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG --- http response --- 05:02:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 ERROR --- http response --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 DEBUG --- status code --- 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362121" 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG { 05:02:01 DEBUG { 05:02:01 DEBUG ----- output ----- 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG { 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG { 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG "data": { 05:02:01 DEBUG "data": { 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG "data": { 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG "data": { 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG "result": [ 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG "result": [ 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:02:01 DEBUG "result": [ 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:02:01 DEBUG { 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG { 05:02:01 DEBUG "result": [ 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG { 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG "metric": { 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG "metric": { 05:02:01 DEBUG { 05:02:01 DEBUG { 05:02:01 DEBUG { 05:02:01 DEBUG "metric": { 05:02:01 DEBUG { 05:02:01 DEBUG "instance": "10.142.0.11:9100" 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG "instance": "10.142.0.11:9100" 05:02:01 DEBUG "metric": { 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG "instance": "10.142.0.11:9100" 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG }, 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:02:01 DEBUG }, 05:02:01 DEBUG "pod": "ds-idrepo-0" 05:02:01 DEBUG "data": { 05:02:01 DEBUG "data": { 05:02:01 DEBUG }, 05:02:01 DEBUG "data": { 05:02:01 DEBUG "value": [ 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG "value": [ 05:02:01 DEBUG }, 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG "value": [ 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG 05:02:01 DEBUG { 05:02:01 DEBUG 05:02:01 DEBUG "value": [ 05:02:01 DEBUG "result": [ 05:02:01 DEBUG "result": [ 05:02:01 DEBUG 05:02:01 DEBUG "result": [ 05:02:01 DEBUG 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG { 05:02:01 DEBUG { 05:02:01 DEBUG 05:02:01 DEBUG { 05:02:01 DEBUG 05:02:01 DEBUG "data": { 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG 05:02:01 DEBUG "metric": { 05:02:01 DEBUG "metric": { 05:02:01 DEBUG 05:02:01 DEBUG "metric": { 05:02:01 DEBUG [...] 05:02:01 DEBUG 05:02:01 DEBUG "result": [ 05:02:01 DEBUG 05:02:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:02:01 DEBUG "pod": "ds-idrepo-0" 05:02:01 DEBUG [...] 05:02:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:02:01 DEBUG 05:02:01 DEBUG [...] 05:02:01 DEBUG { 05:02:01 DEBUG [...] 05:02:01 DEBUG }, 05:02:01 DEBUG }, 05:02:01 DEBUG 05:02:01 DEBUG }, 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG "metric": { 05:02:01 DEBUG 05:02:01 DEBUG "value": [ 05:02:01 DEBUG "value": [ 05:02:01 DEBUG 05:02:01 DEBUG "value": [ 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG "instance": "10.142.0.51:9100" 05:02:01 DEBUG "instance": "10.142.0.11:9100" 05:02:01 DEBUG "instance": "10.142.0.51:9100" 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG }, 05:02:01 DEBUG }, 05:02:01 DEBUG }, 05:02:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:02:01 DEBUG 05:02:01 DEBUG "instance": "10.142.0.51:9100" 05:02:01 DEBUG [...] 05:02:01 DEBUG 05:02:01 DEBUG "value": [ 05:02:01 DEBUG "value": [ 05:02:01 DEBUG "value": [ 05:02:01 DEBUG }, 05:02:01 DEBUG [...] 05:02:01 DEBUG }, 05:02:01 DEBUG 05:02:01 DEBUG [...] 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG "value": [ 05:02:01 DEBUG 05:02:01 DEBUG "value": [ 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG "0.00039999999999812045" 05:02:01 DEBUG 05:02:01 DEBUG "2392827.566666668" 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG ] 05:02:01 DEBUG 05:02:01 DEBUG ] 05:02:01 DEBUG "831000576" 05:02:01 DEBUG 05:02:01 DEBUG "24302.933333333334" 05:02:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:02:01 DEBUG 05:02:01 DEBUG } 05:02:01 DEBUG [...] 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:02:01 DEBUG ] 05:02:01 DEBUG }, 05:02:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:02:01 DEBUG ] 05:02:01 DEBUG 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG }, 05:02:01 DEBUG } 05:02:01 DEBUG "value": [ 05:02:01 DEBUG }, 05:02:01 DEBUG } 05:02:01 DEBUG 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG "value": [ 05:02:01 DEBUG ] 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG "value": [ 05:02:01 DEBUG } 05:02:01 DEBUG 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG } 05:02:01 DEBUG "840.4368291602914" 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG "instance": "10.142.0.51:9100" 05:02:01 DEBUG } 05:02:01 DEBUG "0.5242854259088013" 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG "14876471296" 05:02:01 DEBUG }, 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG "value": [ 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG "2375544.400000001" 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362121" 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG { 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG "data": { 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG "result": [ 05:02:01 DEBUG { 05:02:01 DEBUG "metric": { 05:02:01 DEBUG "instance": "10.142.0.11:9100" 05:02:01 DEBUG }, 05:02:01 DEBUG "value": [ 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG [...] 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG "instance": "10.142.0.51:9100" 05:02:01 DEBUG }, 05:02:01 DEBUG "value": [ 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG "0.0003333333333339776" 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362121" 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 INFO [http_cmd]: http status code OK 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG --- status code --- 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG --- http response --- 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:02:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG ----- output ----- 05:02:01 DEBUG { 05:02:01 DEBUG { 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG "status": "success", 05:02:01 DEBUG "data": { 05:02:01 DEBUG "data": { 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG "resultType": "vector", 05:02:01 DEBUG "result": [ 05:02:01 DEBUG "result": [ 05:02:01 DEBUG { 05:02:01 DEBUG { 05:02:01 DEBUG "metric": { 05:02:01 DEBUG "metric": { 05:02:01 DEBUG "pod": "ds-idrepo-0" 05:02:01 DEBUG "pod": "ds-idrepo-0" 05:02:01 DEBUG }, 05:02:01 DEBUG }, 05:02:01 DEBUG "value": [ 05:02:01 DEBUG "value": [ 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG [...] 05:02:01 DEBUG [...] 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG 05:02:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:02:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:02:01 DEBUG }, 05:02:01 DEBUG }, 05:02:01 DEBUG "value": [ 05:02:01 DEBUG "value": [ 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG 1735362121, 05:02:01 DEBUG "0" 05:02:01 DEBUG "0.001150466588631222" 05:02:01 DEBUG ] 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 DEBUG ] 05:02:01 DEBUG ] 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:01 DEBUG } 05:02:30 INFO 05:02:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:02:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:02:30 INFO [loop_until]: OK (rc = 0) 05:02:30 DEBUG --- stdout --- 05:02:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 112m 13374Mi rcs-687d95c5b-qs2xv 162m 425Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:02:30 DEBUG --- stderr --- 05:02:30 DEBUG 05:02:32 INFO 05:02:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:02:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:02:32 INFO [loop_until]: OK (rc = 0) 05:02:32 DEBUG --- stdout --- 05:02:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 237m 1% 3426Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 203m 0% 18871Mi 66% 05:02:32 DEBUG --- stderr --- 05:02:32 DEBUG 05:03:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362180" 05:03:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362180" 05:03:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362180" 05:03:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362180" 05:03:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362180" 05:03:00 INFO [http_cmd]: http status code OK 05:03:00 DEBUG --- status code --- 05:03:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:00 DEBUG --- http response --- 05:03:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:00 INFO [http_cmd]: http status code OK 05:03:00 DEBUG --- status code --- 05:03:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:00 DEBUG --- http response --- 05:03:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:00 INFO [http_cmd]: http status code OK 05:03:00 DEBUG --- status code --- 05:03:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:00 DEBUG --- http response --- 05:03:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:00 INFO [http_cmd]: http status code OK 05:03:00 INFO [http_cmd]: http status code OK 05:03:00 DEBUG --- status code --- 05:03:00 DEBUG --- status code --- 05:03:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:00 DEBUG --- http response --- 05:03:00 DEBUG --- http response --- 05:03:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 DEBUG --- status code --- 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 ERROR [http_cmd]: ERROR 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- http response --- 05:03:01 ERROR --- status code --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 ERROR --- http response --- 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- http response --- 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362181" 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362181" 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG { 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:03:01 DEBUG "data": { 05:03:01 DEBUG ----- output ----- 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG { 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG { 05:03:01 DEBUG "result": [ 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG { 05:03:01 DEBUG "data": { 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG "data": { 05:03:01 DEBUG "metric": { 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG "instance": "10.142.0.11:9100" 05:03:01 DEBUG "result": [ 05:03:01 DEBUG "result": [ 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG }, 05:03:01 DEBUG { 05:03:01 DEBUG { 05:03:01 DEBUG { 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG "data": { 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG "result": [ 05:03:01 DEBUG { 05:03:01 DEBUG "metric": { 05:03:01 DEBUG "metric": { 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG "value": [ 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG "metric": { 05:03:01 DEBUG "pod": "ds-idrepo-0" 05:03:01 DEBUG "pod": "ds-idrepo-0" 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- http response --- 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362181" 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG "instance": "10.142.0.11:9100" 05:03:01 DEBUG }, 05:03:01 DEBUG }, 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG }, 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "value": [ 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG "value": [ 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG { 05:03:01 DEBUG { 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:03:01 DEBUG [...] 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG "data": { 05:03:01 DEBUG "data": { 05:03:01 DEBUG { 05:03:01 DEBUG 05:03:01 DEBUG { 05:03:01 DEBUG 05:03:01 DEBUG [...] 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:03:01 DEBUG 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG [...] 05:03:01 DEBUG [...] 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG }, 05:03:01 DEBUG "instance": "10.142.0.51:9100" 05:03:01 DEBUG "data": { 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG "result": [ 05:03:01 DEBUG "result": [ 05:03:01 DEBUG "data": { 05:03:01 DEBUG "value": [ 05:03:01 DEBUG }, 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG { 05:03:01 DEBUG { 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "result": [ 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG "metric": { 05:03:01 DEBUG "metric": { 05:03:01 DEBUG "result": [ 05:03:01 DEBUG "505.9639281670032" 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG { 05:03:01 DEBUG "instance": "10.142.0.51:9100" 05:03:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:03:01 DEBUG "instance": "10.142.0.11:9100" 05:03:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:03:01 DEBUG { 05:03:01 DEBUG ] 05:03:01 DEBUG "0.0004333333333306655" 05:03:01 DEBUG "metric": { 05:03:01 DEBUG }, 05:03:01 DEBUG }, 05:03:01 DEBUG }, 05:03:01 DEBUG }, 05:03:01 DEBUG "metric": { 05:03:01 DEBUG } 05:03:01 DEBUG ] 05:03:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "instance": "10.142.0.11:9100" 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG ] 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG }, 05:03:01 DEBUG } 05:03:01 DEBUG }, 05:03:01 DEBUG } 05:03:01 DEBUG "2391938.833333334" 05:03:01 DEBUG "831000576" 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 DEBUG "value": [ 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG "value": [ 05:03:01 DEBUG } 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG 05:03:01 DEBUG [...] 05:03:01 DEBUG [...] 05:03:01 DEBUG 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG [...] 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG [...] 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:03:01 DEBUG 05:03:01 DEBUG "instance": "10.142.0.51:9100" 05:03:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:03:01 DEBUG 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG 05:03:01 DEBUG }, 05:03:01 DEBUG }, 05:03:01 DEBUG 05:03:01 DEBUG { 05:03:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "value": [ 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 DEBUG "instance": "10.142.0.51:9100" 05:03:01 DEBUG }, 05:03:01 DEBUG "value": [ 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG "2378631.5000000005" 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 DEBUG 1735362181, 05:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362181" 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG }, 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG "14877564928" 05:03:01 DEBUG "data": { 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "27170.13333333334" 05:03:01 DEBUG ] 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG "result": [ 05:03:01 DEBUG "0.13747475506438617" 05:03:01 DEBUG } 05:03:01 DEBUG ] 05:03:01 DEBUG { 05:03:01 DEBUG ] 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG "metric": { 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 DEBUG "instance": "10.142.0.11:9100" 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG }, 05:03:01 DEBUG } 05:03:01 DEBUG "value": [ 05:03:01 DEBUG } 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG [...] 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG "instance": "10.142.0.51:9100" 05:03:01 DEBUG }, 05:03:01 DEBUG "value": [ 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG "0.00033333333333303023" 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 INFO [http_cmd]: http status code OK 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG --- status code --- 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG --- http response --- 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:03:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG ----- output ----- 05:03:01 DEBUG { 05:03:01 DEBUG { 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG "status": "success", 05:03:01 DEBUG "data": { 05:03:01 DEBUG "data": { 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG "resultType": "vector", 05:03:01 DEBUG "result": [ 05:03:01 DEBUG "result": [ 05:03:01 DEBUG { 05:03:01 DEBUG { 05:03:01 DEBUG "metric": { 05:03:01 DEBUG "metric": { 05:03:01 DEBUG "pod": "ds-idrepo-0" 05:03:01 DEBUG "pod": "ds-idrepo-0" 05:03:01 DEBUG }, 05:03:01 DEBUG }, 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "value": [ 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG [...] 05:03:01 DEBUG [...] 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG 05:03:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:03:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:03:01 DEBUG }, 05:03:01 DEBUG }, 05:03:01 DEBUG "value": [ 05:03:01 DEBUG "value": [ 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG 1735362181, 05:03:01 DEBUG "0" 05:03:01 DEBUG "0.001203636221617101" 05:03:01 DEBUG ] 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 DEBUG ] 05:03:01 DEBUG ] 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:01 DEBUG } 05:03:30 INFO 05:03:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:03:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:03:31 INFO [loop_until]: OK (rc = 0) 05:03:31 DEBUG --- stdout --- 05:03:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 104m 13374Mi rcs-687d95c5b-qs2xv 123m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:03:31 DEBUG --- stderr --- 05:03:31 DEBUG 05:03:32 INFO 05:03:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:03:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:03:32 INFO [loop_until]: OK (rc = 0) 05:03:32 DEBUG --- stdout --- 05:03:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 209m 1% 3434Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 161m 0% 18872Mi 66% 05:03:32 DEBUG --- stderr --- 05:03:32 DEBUG 05:04:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362240" 05:04:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362240" 05:04:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362240" 05:04:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362240" 05:04:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362240" 05:04:00 INFO [http_cmd]: http status code OK 05:04:00 DEBUG --- status code --- 05:04:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:00 DEBUG --- http response --- 05:04:00 INFO [http_cmd]: http status code OK 05:04:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:00 INFO [http_cmd]: http status code OK 05:04:00 DEBUG --- status code --- 05:04:00 DEBUG --- status code --- 05:04:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:00 DEBUG --- http response --- 05:04:00 DEBUG --- http response --- 05:04:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:00 INFO [http_cmd]: http status code OK 05:04:00 INFO [http_cmd]: http status code OK 05:04:00 DEBUG --- status code --- 05:04:00 DEBUG --- status code --- 05:04:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:00 DEBUG --- http response --- 05:04:00 DEBUG --- http response --- 05:04:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 ERROR [http_cmd]: ERROR 05:04:01 DEBUG --- status code --- 05:04:01 ERROR --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 ERROR --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:04:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362241" 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG { 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG "data": { 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG "result": [ 05:04:01 DEBUG { 05:04:01 DEBUG "metric": { 05:04:01 DEBUG "pod": "ds-idrepo-0" 05:04:01 DEBUG }, 05:04:01 DEBUG "value": [ 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG [...] 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:04:01 DEBUG }, 05:04:01 DEBUG "value": [ 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG "831000576" 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362241" 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362241" 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362241" 05:04:01 DEBUG --- status code --- 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- status code --- 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362241" 05:04:01 DEBUG --- http response --- 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG { 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG "status": "success", 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:04:01 DEBUG { 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG "data": { 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG "data": { 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG { 05:04:01 DEBUG "data": { 05:04:01 DEBUG { 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:04:01 DEBUG "result": [ 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG "result": [ 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG { 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG "data": { 05:04:01 DEBUG "result": [ 05:04:01 DEBUG "data": { 05:04:01 DEBUG { 05:04:01 DEBUG { 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:04:01 DEBUG "metric": { 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG { 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG "metric": { 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG "pod": "ds-idrepo-0" 05:04:01 DEBUG { 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG "result": [ 05:04:01 INFO [http_cmd]: http status code OK 05:04:01 DEBUG "metric": { 05:04:01 DEBUG "result": [ 05:04:01 DEBUG "instance": "10.142.0.11:9100" 05:04:01 DEBUG }, 05:04:01 DEBUG "value": [ 05:04:01 DEBUG }, 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG { 05:04:01 DEBUG --- status code --- 05:04:01 DEBUG "instance": "10.142.0.11:9100" 05:04:01 DEBUG { 05:04:01 DEBUG "data": { 05:04:01 DEBUG { 05:04:01 DEBUG 05:04:01 DEBUG "value": [ 05:04:01 DEBUG "data": { 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:04:01 DEBUG "metric": { 05:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:04:01 DEBUG }, 05:04:01 DEBUG "metric": { 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG "instance": "10.142.0.11:9100" 05:04:01 DEBUG --- http response --- 05:04:01 DEBUG "value": [ 05:04:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:04:01 DEBUG "result": [ 05:04:01 DEBUG "data": { 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "result": [ 05:04:01 DEBUG { 05:04:01 DEBUG }, 05:04:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:04:01 DEBUG 05:04:01 DEBUG }, 05:04:01 DEBUG { 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG [...] 05:04:01 DEBUG 05:04:01 DEBUG { 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG "value": [ 05:04:01 DEBUG ----- output ----- 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "value": [ 05:04:01 DEBUG "result": [ 05:04:01 DEBUG 05:04:01 DEBUG [...] 05:04:01 DEBUG "metric": { 05:04:01 DEBUG "data": { 05:04:01 DEBUG 05:04:01 DEBUG { 05:04:01 DEBUG "metric": { 05:04:01 DEBUG [...] 05:04:01 DEBUG 05:04:01 DEBUG { 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG 05:04:01 DEBUG "status": "success", 05:04:01 DEBUG "instance": "10.142.0.11:9100" 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "metric": { 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "result": [ 05:04:01 DEBUG 05:04:01 DEBUG "data": { 05:04:01 DEBUG }, 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "instance": "10.142.0.11:9100" 05:04:01 DEBUG "instance": "10.142.0.51:9100" 05:04:01 DEBUG }, 05:04:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:04:01 DEBUG { 05:04:01 DEBUG [...] 05:04:01 DEBUG "resultType": "vector", 05:04:01 DEBUG "value": [ 05:04:01 DEBUG 05:04:01 DEBUG [...] 05:04:01 DEBUG }, 05:04:01 DEBUG }, 05:04:01 DEBUG "value": [ 05:04:01 DEBUG }, 05:04:01 DEBUG "metric": { 05:04:01 DEBUG 05:04:01 DEBUG "result": [ 05:04:01 DEBUG 05:04:01 DEBUG "instance": "10.142.0.51:9100" 05:04:01 DEBUG 05:04:01 DEBUG "value": [ 05:04:01 DEBUG "value": [ 05:04:01 DEBUG 05:04:01 DEBUG "value": [ 05:04:01 DEBUG "pod": "ds-idrepo-0" 05:04:01 DEBUG 05:04:01 DEBUG { 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG }, 05:04:01 DEBUG 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG }, 05:04:01 DEBUG "value": [ 05:04:01 DEBUG "metric": { 05:04:01 DEBUG [...] 05:04:01 DEBUG 05:04:01 DEBUG "value": [ 05:04:01 DEBUG 05:04:01 DEBUG "1688662.200000001" 05:04:01 DEBUG 05:04:01 DEBUG "805.9655867647289" 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "pod": "ds-idrepo-0" 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG 05:04:01 DEBUG ] 05:04:01 DEBUG [...] 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG 05:04:01 DEBUG }, 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:04:01 DEBUG [...] 05:04:01 DEBUG } 05:04:01 DEBUG 05:04:01 DEBUG "instance": "10.142.0.51:9100" 05:04:01 DEBUG ] 05:04:01 DEBUG 05:04:01 DEBUG "value": [ 05:04:01 DEBUG "0.0002666666666717296" 05:04:01 DEBUG "instance": "10.142.0.51:9100" 05:04:01 DEBUG }, 05:04:01 DEBUG 05:04:01 DEBUG ] 05:04:01 DEBUG 05:04:01 DEBUG }, 05:04:01 DEBUG } 05:04:01 DEBUG [...] 05:04:01 DEBUG 05:04:01 DEBUG ] 05:04:01 DEBUG }, 05:04:01 DEBUG "value": [ 05:04:01 DEBUG 05:04:01 DEBUG } 05:04:01 DEBUG 05:04:01 DEBUG "value": [ 05:04:01 DEBUG } 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG 05:04:01 DEBUG } 05:04:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG "value": [ 05:04:01 DEBUG } 05:04:01 DEBUG "15428.26666666667" 05:04:01 DEBUG "instance": "10.142.0.51:9100" 05:04:01 DEBUG }, 05:04:01 DEBUG "1671726.0333333334" 05:04:01 DEBUG 05:04:01 DEBUG [...] 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG }, 05:04:01 DEBUG "value": [ 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:04:01 DEBUG 05:04:01 DEBUG "14876598272" 05:04:01 DEBUG } 05:04:01 DEBUG "value": [ 05:04:01 DEBUG ] 05:04:01 DEBUG "0.07729614392712983" 05:04:01 DEBUG }, 05:04:01 DEBUG 05:04:01 DEBUG 05:04:01 DEBUG ] 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG "0" 05:04:01 DEBUG ] 05:04:01 DEBUG "value": [ 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG "0" 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG }, 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:01 DEBUG "value": [ 05:04:01 DEBUG ] 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG 1735362241, 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:01 DEBUG "0.0011749605447405084" 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG ] 05:04:01 DEBUG } 05:04:01 DEBUG } 05:04:31 INFO 05:04:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:04:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:04:31 INFO [loop_until]: OK (rc = 0) 05:04:31 DEBUG --- stdout --- 05:04:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 106m 13374Mi rcs-687d95c5b-qs2xv 109m 425Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:04:31 DEBUG --- stderr --- 05:04:31 DEBUG 05:04:32 INFO 05:04:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:04:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:04:32 INFO [loop_until]: OK (rc = 0) 05:04:32 DEBUG --- stdout --- 05:04:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 179m 1% 3471Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 117m 0% 18874Mi 66% 05:04:32 DEBUG --- stderr --- 05:04:32 DEBUG 05:05:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362300" 05:05:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362300" 05:05:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362300" 05:05:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362300" 05:05:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362300" 05:05:00 INFO [http_cmd]: http status code OK 05:05:00 DEBUG --- status code --- 05:05:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:00 DEBUG --- http response --- 05:05:00 INFO [http_cmd]: http status code OK 05:05:00 INFO [http_cmd]: http status code OK 05:05:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:00 INFO [http_cmd]: http status code OK 05:05:00 DEBUG --- status code --- 05:05:00 INFO [http_cmd]: http status code OK 05:05:00 DEBUG --- status code --- 05:05:00 DEBUG --- status code --- 05:05:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:00 DEBUG --- status code --- 05:05:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:00 DEBUG --- http response --- 05:05:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:00 DEBUG --- http response --- 05:05:00 DEBUG --- http response --- 05:05:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:00 DEBUG --- http response --- 05:05:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 ERROR [http_cmd]: ERROR 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- status code --- 05:05:01 ERROR --- status code --- 05:05:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:05:01 ERROR --- http response --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG --- http response --- 05:05:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG { 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG "data": { 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG "result": [ 05:05:01 DEBUG { 05:05:01 DEBUG "metric": { 05:05:01 DEBUG "pod": "ds-idrepo-0" 05:05:01 DEBUG }, 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG [...] 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:05:01 DEBUG }, 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG "831000576" 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362301" 05:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362301" 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG { 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:05:01 DEBUG "data": { 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG "resultType": "vector", 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG { 05:05:01 DEBUG "result": [ 05:05:01 DEBUG { 05:05:01 DEBUG "metric": { 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG "status": "success", 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG "instance": "10.142.0.11:9100" 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG "data": { 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG }, 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG "value": [ 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG "result": [ 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:05:01 DEBUG 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG { 05:05:01 DEBUG { 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:05:01 DEBUG { 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG "metric": { 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG { 05:05:01 DEBUG 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG "data": { 05:05:01 DEBUG "instance": "10.142.0.11:9100" 05:05:01 DEBUG { 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG [...] 05:05:01 DEBUG 05:05:01 DEBUG { 05:05:01 DEBUG "data": { 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG }, 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG "data": { 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:05:01 DEBUG 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG "result": [ 05:05:01 DEBUG "value": [ 05:05:01 DEBUG "data": { 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG 05:05:01 DEBUG "data": { 05:05:01 DEBUG "result": [ 05:05:01 DEBUG { 05:05:01 DEBUG 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG "result": [ 05:05:01 DEBUG { 05:05:01 DEBUG "instance": "10.142.0.51:9100" 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG { 05:05:01 DEBUG "metric": { 05:05:01 DEBUG 05:05:01 DEBUG "result": [ 05:05:01 DEBUG { 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG }, 05:05:01 DEBUG "result": [ 05:05:01 DEBUG "metric": { 05:05:01 DEBUG "instance": "10.142.0.11:9100" 05:05:01 DEBUG 05:05:01 DEBUG { 05:05:01 DEBUG "metric": { 05:05:01 DEBUG "data": { 05:05:01 DEBUG "value": [ 05:05:01 DEBUG { 05:05:01 DEBUG "pod": "ds-idrepo-0" 05:05:01 DEBUG }, 05:05:01 DEBUG [...] 05:05:01 DEBUG "metric": { 05:05:01 DEBUG "instance": "10.142.0.11:9100" 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG "metric": { 05:05:01 DEBUG }, 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 05:05:01 DEBUG "pod": "ds-idrepo-0" 05:05:01 DEBUG }, 05:05:01 DEBUG "result": [ 05:05:01 DEBUG "0.0004333333333306656" 05:05:01 DEBUG "instance": "10.142.0.11:9100" 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 INFO [http_cmd]: http status code OK 05:05:01 DEBUG 05:05:01 DEBUG }, 05:05:01 DEBUG "value": [ 05:05:01 DEBUG { 05:05:01 DEBUG ] 05:05:01 DEBUG }, 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG --- status code --- 05:05:01 DEBUG 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 05:05:01 DEBUG "metric": { 05:05:01 DEBUG } 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:05:01 DEBUG "instance": "10.142.0.51:9100" 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:05:01 DEBUG ] 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG [...] 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG --- http response --- 05:05:01 DEBUG }, 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG }, 05:05:01 DEBUG } 05:05:01 DEBUG 05:05:01 DEBUG [...] 05:05:01 DEBUG 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:05:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 05:05:01 DEBUG [...] 05:05:01 DEBUG "value": [ 05:05:01 DEBUG } 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG ----- output ----- 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG [...] 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG [...] 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG { 05:05:01 DEBUG { 05:05:01 DEBUG "24712.53333333334" 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG "instance": "10.142.0.51:9100" 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG "status": "success", 05:05:01 DEBUG ] 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:05:01 DEBUG }, 05:05:01 DEBUG "data": { 05:05:01 DEBUG "data": { 05:05:01 DEBUG } 05:05:01 DEBUG 05:05:01 DEBUG "instance": "10.142.0.51:9100" 05:05:01 DEBUG [...] 05:05:01 DEBUG 05:05:01 DEBUG }, 05:05:01 DEBUG "value": [ 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG "resultType": "vector", 05:05:01 DEBUG ] 05:05:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:05:01 DEBUG }, 05:05:01 DEBUG 05:05:01 DEBUG "instance": "10.142.0.51:9100" 05:05:01 DEBUG }, 05:05:01 DEBUG "value": [ 05:05:01 DEBUG "result": [ 05:05:01 DEBUG "result": [ 05:05:01 DEBUG } 05:05:01 DEBUG }, 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG { 05:05:01 DEBUG { 05:05:01 DEBUG } 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG "0.0003333333333335039" 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG ] 05:05:01 DEBUG "metric": { 05:05:01 DEBUG "metric": { 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG "1248536.5666666671" 05:05:01 DEBUG "1233882.9333333336" 05:05:01 DEBUG ] 05:05:01 DEBUG "pod": "ds-idrepo-0" 05:05:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:05:01 DEBUG "0" 05:05:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:05:01 DEBUG "739.421139017852" 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:01 DEBUG }, 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:01 DEBUG }, 05:05:01 DEBUG }, 05:05:01 DEBUG ] 05:05:01 DEBUG "value": [ 05:05:01 DEBUG } 05:05:01 DEBUG ] 05:05:01 DEBUG "value": [ 05:05:01 DEBUG "value": [ 05:05:01 DEBUG } 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG ] 05:05:01 DEBUG "0.03568432629068357" 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG } 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:01 DEBUG [...] 05:05:01 DEBUG [...] 05:05:01 DEBUG ] 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG } 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG } 05:05:01 DEBUG 05:05:01 DEBUG 05:05:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:05:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:05:01 DEBUG }, 05:05:01 DEBUG }, 05:05:01 DEBUG "value": [ 05:05:01 DEBUG "value": [ 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG 1735362301, 05:05:01 DEBUG "0.001135575125799922" 05:05:01 DEBUG "14878736384" 05:05:01 DEBUG ] 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:01 DEBUG ] 05:05:01 DEBUG ] 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:01 DEBUG } 05:05:31 INFO 05:05:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:05:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:05:31 INFO [loop_until]: OK (rc = 0) 05:05:31 DEBUG --- stdout --- 05:05:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 81m 13374Mi rcs-687d95c5b-qs2xv 87m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:05:31 DEBUG --- stderr --- 05:05:31 DEBUG 05:05:32 INFO 05:05:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:05:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:05:33 INFO [loop_until]: OK (rc = 0) 05:05:33 DEBUG --- stdout --- 05:05:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 179m 1% 3449Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 142m 0% 18875Mi 66% 05:05:33 DEBUG --- stderr --- 05:05:33 DEBUG 05:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362360" 05:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362360" 05:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362360" 05:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362360" 05:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362360" 05:06:00 INFO [http_cmd]: http status code OK 05:06:00 DEBUG --- status code --- 05:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:00 DEBUG --- http response --- 05:06:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:00 INFO [http_cmd]: http status code OK 05:06:00 DEBUG --- status code --- 05:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:00 DEBUG --- http response --- 05:06:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:00 INFO [http_cmd]: http status code OK 05:06:00 DEBUG --- status code --- 05:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:00 DEBUG --- http response --- 05:06:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:00 INFO [http_cmd]: http status code OK 05:06:00 DEBUG --- status code --- 05:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:00 DEBUG --- http response --- 05:06:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:00 INFO [http_cmd]: http status code OK 05:06:00 DEBUG --- status code --- 05:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:00 DEBUG --- http response --- 05:06:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 ERROR [http_cmd]: ERROR 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- http response --- 05:06:01 ERROR --- status code --- 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 ERROR --- http response --- 05:06:01 DEBUG --- http response --- 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG --- status code --- 05:06:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG { 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG "data": { 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG "result": [ 05:06:01 DEBUG { 05:06:01 DEBUG "metric": { 05:06:01 DEBUG "pod": "ds-idrepo-0" 05:06:01 DEBUG }, 05:06:01 DEBUG "value": [ 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG [...] 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:06:01 DEBUG }, 05:06:01 DEBUG "value": [ 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG "831000576" 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG } 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362361" 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG { 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG "data": { 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG "result": [ 05:06:01 DEBUG { 05:06:01 DEBUG "metric": { 05:06:01 DEBUG "instance": "10.142.0.11:9100" 05:06:01 DEBUG }, 05:06:01 DEBUG "value": [ 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG [...] 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG "instance": "10.142.0.51:9100" 05:06:01 DEBUG }, 05:06:01 DEBUG "value": [ 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG "0.0004666666666698424" 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG } 05:06:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362361" 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG { 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG "data": { 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG "result": [ 05:06:01 DEBUG { 05:06:01 DEBUG "metric": { 05:06:01 DEBUG "instance": "10.142.0.11:9100" 05:06:01 DEBUG }, 05:06:01 DEBUG "value": [ 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG [...] 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG "instance": "10.142.0.51:9100" 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:06:01 DEBUG }, 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:06:01 DEBUG "value": [ 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG { 05:06:01 DEBUG { 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG { 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:06:01 DEBUG { 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG "data": { 05:06:01 DEBUG "data": { 05:06:01 DEBUG "19660.800000000003" 05:06:01 DEBUG "data": { 05:06:01 DEBUG { 05:06:01 DEBUG { 05:06:01 DEBUG "data": { 05:06:01 DEBUG { 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG ] 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG "result": [ 05:06:01 DEBUG "result": [ 05:06:01 DEBUG } 05:06:01 DEBUG "result": [ 05:06:01 DEBUG "data": { 05:06:01 DEBUG "data": { 05:06:01 DEBUG "result": [ 05:06:01 DEBUG "data": { 05:06:01 DEBUG { 05:06:01 DEBUG { 05:06:01 DEBUG ] 05:06:01 DEBUG { 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG { 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG "metric": { 05:06:01 DEBUG "metric": { 05:06:01 DEBUG } 05:06:01 DEBUG "metric": { 05:06:01 DEBUG "result": [ 05:06:01 DEBUG "result": [ 05:06:01 DEBUG "metric": { 05:06:01 DEBUG "result": [ 05:06:01 DEBUG "instance": "10.142.0.11:9100" 05:06:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:06:01 DEBUG } 05:06:01 DEBUG "instance": "10.142.0.11:9100" 05:06:01 DEBUG { 05:06:01 DEBUG { 05:06:01 DEBUG "pod": "ds-cts-0" 05:06:01 DEBUG { 05:06:01 DEBUG }, 05:06:01 DEBUG }, 05:06:01 DEBUG }, 05:06:01 DEBUG "metric": { 05:06:01 DEBUG "metric": { 05:06:01 DEBUG }, 05:06:01 DEBUG "metric": { 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "instance": "10.142.0.11:9100" 05:06:01 DEBUG "pod": "ds-cts-0" 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "pod": "ds-cts-0" 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG }, 05:06:01 DEBUG }, 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "value": [ 05:06:01 DEBUG }, 05:06:01 DEBUG 05:06:01 INFO [http_cmd]: http status code OK 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG "value": [ 05:06:01 DEBUG [...] 05:06:01 DEBUG --- status code --- 05:06:01 DEBUG [...] 05:06:01 DEBUG [...] 05:06:01 DEBUG [...] 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG [...] 05:06:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG --- http response --- 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG [...] 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG ----- output ----- 05:06:01 DEBUG "instance": "10.142.0.51:9100" 05:06:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:06:01 DEBUG "instance": "10.142.0.51:9100" 05:06:01 DEBUG 05:06:01 DEBUG [...] 05:06:01 DEBUG "pod": "ds-idrepo-0" 05:06:01 DEBUG "pod": "ds-idrepo-0" 05:06:01 DEBUG { 05:06:01 DEBUG }, 05:06:01 DEBUG }, 05:06:01 DEBUG }, 05:06:01 DEBUG 05:06:01 DEBUG 05:06:01 DEBUG }, 05:06:01 DEBUG }, 05:06:01 DEBUG "status": "success", 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "instance": "10.142.0.51:9100" 05:06:01 DEBUG 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "data": { 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG }, 05:06:01 DEBUG 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG "resultType": "vector", 05:06:01 DEBUG "80502.73333333335" 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG "value": [ 05:06:01 DEBUG "pod": "ds-idrepo-0" 05:06:01 DEBUG "1448605.9110406018" 05:06:01 DEBUG "0.056372054762229167" 05:06:01 DEBUG "result": [ 05:06:01 DEBUG "0.05064181694897166" 05:06:01 DEBUG "0.0003333333333335039" 05:06:01 DEBUG ] 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG "95684.2" 05:06:01 DEBUG ] 05:06:01 DEBUG ] 05:06:01 DEBUG { 05:06:01 DEBUG ] 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG } 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG } 05:06:01 DEBUG "metric": { 05:06:01 DEBUG } 05:06:01 DEBUG } 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG } 05:06:01 DEBUG ] 05:06:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:06:01 DEBUG ] 05:06:01 DEBUG }, 05:06:01 DEBUG } 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG }, 05:06:01 DEBUG } 05:06:01 DEBUG "value": [ 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG } 05:06:01 DEBUG "value": [ 05:06:01 DEBUG } 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG } 05:06:01 DEBUG } 05:06:01 DEBUG 05:06:01 DEBUG "962.6887596215995" 05:06:01 DEBUG } 05:06:01 DEBUG 05:06:01 DEBUG ] 05:06:01 DEBUG 05:06:01 DEBUG [...] 05:06:01 DEBUG 05:06:01 DEBUG } 05:06:01 DEBUG 05:06:01 DEBUG ] 05:06:01 DEBUG 05:06:01 DEBUG } 05:06:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:06:01 DEBUG } 05:06:01 DEBUG }, 05:06:01 DEBUG "value": [ 05:06:01 DEBUG 1735362361, 05:06:01 DEBUG "14881476608" 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG ] 05:06:01 DEBUG } 05:06:01 DEBUG } 05:06:31 INFO 05:06:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:06:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:06:31 INFO [loop_until]: OK (rc = 0) 05:06:31 DEBUG --- stdout --- 05:06:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 62m 13375Mi rcs-687d95c5b-qs2xv 30m 426Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:06:31 DEBUG --- stderr --- 05:06:31 DEBUG 05:06:33 INFO 05:06:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:06:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:06:33 INFO [loop_until]: OK (rc = 0) 05:06:33 DEBUG --- stdout --- 05:06:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 136m 0% 3451Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 125m 0% 18875Mi 66% 05:06:33 DEBUG --- stderr --- 05:06:33 DEBUG 05:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362420" 05:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362420" 05:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362420" 05:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362420" 05:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362420" 05:07:00 INFO [http_cmd]: http status code OK 05:07:00 DEBUG --- status code --- 05:07:00 INFO [http_cmd]: http status code OK 05:07:00 INFO [http_cmd]: http status code OK 05:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:00 INFO [http_cmd]: http status code OK 05:07:00 INFO [http_cmd]: http status code OK 05:07:00 DEBUG --- status code --- 05:07:00 DEBUG --- status code --- 05:07:00 DEBUG --- http response --- 05:07:00 DEBUG --- status code --- 05:07:00 DEBUG --- status code --- 05:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:00 DEBUG --- http response --- 05:07:00 DEBUG --- http response --- 05:07:00 DEBUG --- http response --- 05:07:00 DEBUG --- http response --- 05:07:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 ERROR [http_cmd]: ERROR 05:07:01 DEBUG --- status code --- 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 ERROR --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 ERROR --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- status code --- 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- http response --- 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG { 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG "data": { 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG "result": [ 05:07:01 DEBUG { 05:07:01 DEBUG "metric": { 05:07:01 DEBUG "pod": "ds-idrepo-0" 05:07:01 DEBUG }, 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG [...] 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:07:01 DEBUG }, 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG "831000576" 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362421" 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG { 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG "data": { 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG "result": [ 05:07:01 DEBUG { 05:07:01 DEBUG "metric": { 05:07:01 DEBUG "instance": "10.142.0.11:9100" 05:07:01 DEBUG }, 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG [...] 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG "instance": "10.142.0.51:9100" 05:07:01 DEBUG }, 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG "0.001733333333334031" 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362421" 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- status code --- 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362421" 05:07:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362421" 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG { 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG "data": { 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG "result": [ 05:07:01 DEBUG { 05:07:01 DEBUG { 05:07:01 DEBUG "status": "success", 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG "metric": { 05:07:01 DEBUG "data": { 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG "instance": "10.142.0.11:9100" 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG }, 05:07:01 DEBUG "result": [ 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG "value": [ 05:07:01 DEBUG { 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG 05:07:01 DEBUG "metric": { 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG 05:07:01 DEBUG "instance": "10.142.0.11:9100" 05:07:01 DEBUG { 05:07:01 DEBUG { 05:07:01 DEBUG { 05:07:01 INFO [http_cmd]: http status code OK 05:07:01 DEBUG { 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:07:01 DEBUG 05:07:01 DEBUG }, 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG --- status code --- 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG [...] 05:07:01 DEBUG "value": [ 05:07:01 DEBUG "data": { 05:07:01 DEBUG "data": { 05:07:01 DEBUG "data": { 05:07:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:07:01 DEBUG "data": { 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:07:01 DEBUG { 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG --- http response --- 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG "result": [ 05:07:01 DEBUG "result": [ 05:07:01 DEBUG "result": [ 05:07:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:07:01 DEBUG "result": [ 05:07:01 DEBUG { 05:07:01 DEBUG "data": { 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG { 05:07:01 DEBUG { 05:07:01 DEBUG { 05:07:01 DEBUG ----- output ----- 05:07:01 DEBUG { 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG "instance": "10.142.0.51:9100" 05:07:01 DEBUG [...] 05:07:01 DEBUG "metric": { 05:07:01 DEBUG "metric": { 05:07:01 DEBUG "metric": { 05:07:01 DEBUG { 05:07:01 DEBUG "metric": { 05:07:01 DEBUG "data": { 05:07:01 DEBUG "result": [ 05:07:01 DEBUG }, 05:07:01 DEBUG 05:07:01 DEBUG "instance": "10.142.0.11:9100" 05:07:01 DEBUG "pod": "ds-idrepo-0" 05:07:01 DEBUG "instance": "10.142.0.11:9100" 05:07:01 DEBUG "status": "success", 05:07:01 DEBUG "pod": "ds-idrepo-0" 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG { 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 05:07:01 DEBUG }, 05:07:01 DEBUG }, 05:07:01 DEBUG }, 05:07:01 DEBUG "data": { 05:07:01 DEBUG }, 05:07:01 DEBUG "result": [ 05:07:01 DEBUG "metric": { 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG 05:07:01 DEBUG "value": [ 05:07:01 DEBUG "value": [ 05:07:01 DEBUG "value": [ 05:07:01 DEBUG "resultType": "vector", 05:07:01 DEBUG "value": [ 05:07:01 DEBUG { 05:07:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:07:01 DEBUG "27852.800000000003" 05:07:01 DEBUG "instance": "10.142.0.51:9100" 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG "result": [ 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG ] 05:07:01 DEBUG }, 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG { 05:07:01 DEBUG "metric": { 05:07:01 DEBUG }, 05:07:01 DEBUG [...] 05:07:01 DEBUG } 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG "metric": { 05:07:01 DEBUG "pod": "ds-idrepo-0" 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 05:07:01 DEBUG ] 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG [...] 05:07:01 DEBUG [...] 05:07:01 DEBUG [...] 05:07:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:07:01 DEBUG }, 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG } 05:07:01 DEBUG "1350902.5666666671" 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG }, 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG } 05:07:01 DEBUG ] 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:07:01 DEBUG } 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG [...] 05:07:01 DEBUG }, 05:07:01 DEBUG ] 05:07:01 DEBUG "instance": "10.142.0.51:9100" 05:07:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:07:01 DEBUG "instance": "10.142.0.51:9100" 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG "value": [ 05:07:01 DEBUG } 05:07:01 DEBUG }, 05:07:01 DEBUG }, 05:07:01 DEBUG }, 05:07:01 DEBUG 05:07:01 DEBUG [...] 05:07:01 DEBUG 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG } 05:07:01 DEBUG "value": [ 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG "0.0011014444339265046" 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG "0" 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG "1368197.3" 05:07:01 DEBUG [...] 05:07:01 DEBUG 05:07:01 DEBUG 05:07:01 DEBUG ] 05:07:01 DEBUG ] 05:07:01 DEBUG "value": [ 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG 05:07:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG 05:07:01 DEBUG ] 05:07:01 DEBUG 05:07:01 DEBUG }, 05:07:01 DEBUG } 05:07:01 DEBUG ] 05:07:01 DEBUG "0.0020000000000000764" 05:07:01 DEBUG 05:07:01 DEBUG } 05:07:01 DEBUG 05:07:01 DEBUG "value": [ 05:07:01 DEBUG } 05:07:01 DEBUG ] 05:07:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:07:01 DEBUG } 05:07:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:01 DEBUG }, 05:07:01 DEBUG }, 05:07:01 DEBUG "0.08863966556060947" 05:07:01 DEBUG ] 05:07:01 DEBUG "value": [ 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG "568.8610839658479" 05:07:01 DEBUG ] 05:07:01 DEBUG "value": [ 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:01 DEBUG 1735362421, 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:01 DEBUG ] 05:07:01 DEBUG "14880321536" 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:01 DEBUG ] 05:07:01 DEBUG } 05:07:01 DEBUG } 05:07:31 INFO 05:07:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:07:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:07:31 INFO [loop_until]: OK (rc = 0) 05:07:31 DEBUG --- stdout --- 05:07:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 86m 13376Mi rcs-687d95c5b-qs2xv 130m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:07:31 DEBUG --- stderr --- 05:07:31 DEBUG 05:07:33 INFO 05:07:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:07:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:07:33 INFO [loop_until]: OK (rc = 0) 05:07:33 DEBUG --- stdout --- 05:07:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 224m 1% 3444Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 191m 0% 18879Mi 66% 05:07:33 DEBUG --- stderr --- 05:07:33 DEBUG 05:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362480" 05:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362480" 05:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362480" 05:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362480" 05:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362480" 05:08:00 INFO [http_cmd]: http status code OK 05:08:00 INFO [http_cmd]: http status code OK 05:08:00 INFO [http_cmd]: http status code OK 05:08:00 DEBUG --- status code --- 05:08:00 INFO [http_cmd]: http status code OK 05:08:00 DEBUG --- status code --- 05:08:00 INFO [http_cmd]: http status code OK 05:08:00 DEBUG --- status code --- 05:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:00 DEBUG --- status code --- 05:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:00 DEBUG --- status code --- 05:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:00 DEBUG --- http response --- 05:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:00 DEBUG --- http response --- 05:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:00 DEBUG --- http response --- 05:08:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:00 DEBUG --- http response --- 05:08:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:00 DEBUG --- http response --- 05:08:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 DEBUG --- status code --- 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362481" 05:08:01 DEBUG --- http response --- 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 ERROR [http_cmd]: ERROR 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 ERROR --- status code --- 05:08:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:08:01 ERROR --- http response --- 05:08:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:08:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG { 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG "data": { 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG "result": [ 05:08:01 DEBUG { 05:08:01 DEBUG "metric": { 05:08:01 DEBUG "pod": "ds-idrepo-0" 05:08:01 DEBUG }, 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG [...] 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:08:01 DEBUG }, 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG "831016960" 05:08:01 DEBUG ] 05:08:01 DEBUG } 05:08:01 DEBUG ] 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362481" 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG { 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG "data": { 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG "result": [ 05:08:01 DEBUG { 05:08:01 DEBUG "metric": { 05:08:01 DEBUG "instance": "10.142.0.11:9100" 05:08:01 DEBUG }, 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG [...] 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG "instance": "10.142.0.51:9100" 05:08:01 DEBUG }, 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG "0.0004999999999967032" 05:08:01 DEBUG ] 05:08:01 DEBUG } 05:08:01 DEBUG ] 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362481" 05:08:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362481" 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG { 05:08:01 DEBUG { 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG "data": { 05:08:01 DEBUG "data": { 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG "result": [ 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG "result": [ 05:08:01 DEBUG { 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG { 05:08:01 DEBUG "metric": { 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG "metric": { 05:08:01 DEBUG "instance": "10.142.0.11:9100" 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG "instance": "10.142.0.11:9100" 05:08:01 DEBUG }, 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:08:01 DEBUG }, 05:08:01 DEBUG "value": [ 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 05:08:01 DEBUG { 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG { 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG { 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG "data": { 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:08:01 DEBUG "data": { 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG { 05:08:01 DEBUG 05:08:01 DEBUG [...] 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG "data": { 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG [...] 05:08:01 DEBUG 05:08:01 DEBUG "result": [ 05:08:01 DEBUG { 05:08:01 DEBUG "result": [ 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG "data": { 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 INFO [http_cmd]: http status code OK 05:08:01 DEBUG { 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG { 05:08:01 DEBUG "result": [ 05:08:01 DEBUG { 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG --- status code --- 05:08:01 DEBUG "metric": { 05:08:01 DEBUG "data": { 05:08:01 DEBUG "metric": { 05:08:01 DEBUG { 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG "result": [ 05:08:01 DEBUG 05:08:01 DEBUG "instance": "10.142.0.51:9100" 05:08:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:08:01 DEBUG "pod": "ds-idrepo-0" 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG "instance": "10.142.0.11:9100" 05:08:01 DEBUG "metric": { 05:08:01 DEBUG "data": { 05:08:01 DEBUG { 05:08:01 DEBUG "instance": "10.142.0.51:9100" 05:08:01 DEBUG }, 05:08:01 DEBUG --- http response --- 05:08:01 DEBUG }, 05:08:01 DEBUG "result": [ 05:08:01 DEBUG }, 05:08:01 DEBUG "pod": "ds-idrepo-0" 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG "metric": { 05:08:01 DEBUG }, 05:08:01 DEBUG "value": [ 05:08:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:08:01 DEBUG "value": [ 05:08:01 DEBUG { 05:08:01 DEBUG "value": [ 05:08:01 DEBUG }, 05:08:01 DEBUG "result": [ 05:08:01 DEBUG "instance": "10.142.0.11:9100" 05:08:01 DEBUG }, 05:08:01 DEBUG "value": [ 05:08:01 DEBUG ----- output ----- 05:08:01 DEBUG 05:08:01 DEBUG "metric": { 05:08:01 DEBUG 05:08:01 DEBUG "value": [ 05:08:01 DEBUG { 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG 05:08:01 DEBUG { 05:08:01 DEBUG 05:08:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG "metric": { 05:08:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:08:01 DEBUG "13789.866666666669" 05:08:01 DEBUG 05:08:01 DEBUG "status": "success", 05:08:01 DEBUG 05:08:01 DEBUG }, 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG }, 05:08:01 DEBUG ] 05:08:01 DEBUG 05:08:01 DEBUG "data": { 05:08:01 DEBUG [...] 05:08:01 DEBUG "value": [ 05:08:01 DEBUG [...] 05:08:01 DEBUG 05:08:01 DEBUG "2369271.4000000004" 05:08:01 DEBUG "value": [ 05:08:01 DEBUG } 05:08:01 DEBUG [...] 05:08:01 DEBUG "resultType": "vector", 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG [...] 05:08:01 DEBUG ] 05:08:01 DEBUG 05:08:01 DEBUG ] 05:08:01 DEBUG 05:08:01 DEBUG "result": [ 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG } 05:08:01 DEBUG 05:08:01 DEBUG } 05:08:01 DEBUG 05:08:01 DEBUG { 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG 05:08:01 DEBUG ] 05:08:01 DEBUG 05:08:01 DEBUG } 05:08:01 DEBUG 05:08:01 DEBUG "metric": { 05:08:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:08:01 DEBUG [...] 05:08:01 DEBUG "instance": "10.142.0.51:9100" 05:08:01 DEBUG 05:08:01 DEBUG } 05:08:01 DEBUG [...] 05:08:01 DEBUG "instance": "10.142.0.51:9100" 05:08:01 DEBUG "pod": "ds-idrepo-0" 05:08:01 DEBUG }, 05:08:01 DEBUG 05:08:01 DEBUG }, 05:08:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:08:01 DEBUG } 05:08:01 DEBUG 05:08:01 DEBUG }, 05:08:01 DEBUG }, 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 05:08:01 DEBUG "value": [ 05:08:01 DEBUG }, 05:08:01 DEBUG 05:08:01 DEBUG "value": [ 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG 05:08:01 DEBUG "0" 05:08:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:08:01 DEBUG "2386023.8666666667" 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:08:01 DEBUG "0.00033333333333303023" 05:08:01 DEBUG 05:08:01 DEBUG ] 05:08:01 DEBUG }, 05:08:01 DEBUG ] 05:08:01 DEBUG "0.0011312241934106752" 05:08:01 DEBUG }, 05:08:01 DEBUG ] 05:08:01 DEBUG 05:08:01 DEBUG } 05:08:01 DEBUG "value": [ 05:08:01 DEBUG } 05:08:01 DEBUG ] 05:08:01 DEBUG "value": [ 05:08:01 DEBUG } 05:08:01 DEBUG [...] 05:08:01 DEBUG ] 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG ] 05:08:01 DEBUG } 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG ] 05:08:01 DEBUG 05:08:01 DEBUG } 05:08:01 DEBUG "14882426880" 05:08:01 DEBUG } 05:08:01 DEBUG ] 05:08:01 DEBUG "0.18687039873183836" 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:01 DEBUG ] 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:01 DEBUG ] 05:08:01 DEBUG 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:01 DEBUG 05:08:01 DEBUG ] 05:08:01 DEBUG ] 05:08:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:01 DEBUG }, 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:01 DEBUG "value": [ 05:08:01 DEBUG 1735362481, 05:08:01 DEBUG "262.8505750000455" 05:08:01 DEBUG ] 05:08:01 DEBUG } 05:08:01 DEBUG ] 05:08:01 DEBUG } 05:08:01 DEBUG } 05:08:31 INFO 05:08:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:08:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:08:31 INFO [loop_until]: OK (rc = 0) 05:08:31 DEBUG --- stdout --- 05:08:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 90m 13375Mi rcs-687d95c5b-qs2xv 210m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:08:31 DEBUG --- stderr --- 05:08:31 DEBUG 05:08:33 INFO 05:08:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:08:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:08:33 INFO [loop_until]: OK (rc = 0) 05:08:33 DEBUG --- stdout --- 05:08:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 241m 1% 3456Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 185m 0% 18872Mi 66% 05:08:33 DEBUG --- stderr --- 05:08:33 DEBUG 05:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362540" 05:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362540" 05:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362540" 05:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362540" 05:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362540" 05:09:00 INFO [http_cmd]: http status code OK 05:09:00 DEBUG --- status code --- 05:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:00 DEBUG --- http response --- 05:09:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:00 INFO [http_cmd]: http status code OK 05:09:00 DEBUG --- status code --- 05:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:00 DEBUG --- http response --- 05:09:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:00 INFO [http_cmd]: http status code OK 05:09:00 DEBUG --- status code --- 05:09:00 INFO [http_cmd]: http status code OK 05:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:00 INFO [http_cmd]: http status code OK 05:09:00 DEBUG --- status code --- 05:09:00 DEBUG --- http response --- 05:09:00 DEBUG --- status code --- 05:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:00 DEBUG --- http response --- 05:09:00 DEBUG --- http response --- 05:09:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 ERROR [http_cmd]: ERROR 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 ERROR --- status code --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- status code --- 05:09:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 ERROR --- http response --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362541" 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG { 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG "data": { 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG "result": [ 05:09:01 DEBUG { 05:09:01 DEBUG "metric": { 05:09:01 DEBUG "pod": "ds-idrepo-0" 05:09:01 DEBUG }, 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG [...] 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:09:01 DEBUG }, 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG "831016960" 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG } 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362541" 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG { 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG "data": { 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG "result": [ 05:09:01 DEBUG { 05:09:01 DEBUG "metric": { 05:09:01 DEBUG "instance": "10.142.0.11:9100" 05:09:01 DEBUG }, 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG [...] 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG "instance": "10.142.0.51:9100" 05:09:01 DEBUG }, 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG "0.0011999999999943614" 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG } 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362541" 05:09:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362541" 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG { 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG "data": { 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG "result": [ 05:09:01 DEBUG { 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG "metric": { 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG "instance": "10.142.0.11:9100" 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG }, 05:09:01 DEBUG { 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG "value": [ 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:09:01 DEBUG 05:09:01 DEBUG "data": { 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG { 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG { 05:09:01 DEBUG 05:09:01 DEBUG "result": [ 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG [...] 05:09:01 DEBUG { 05:09:01 DEBUG "data": { 05:09:01 INFO [http_cmd]: http status code OK 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG "data": { 05:09:01 DEBUG 05:09:01 DEBUG "metric": { 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG --- status code --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG 05:09:01 DEBUG "instance": "10.142.0.11:9100" 05:09:01 DEBUG "result": [ 05:09:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:09:01 DEBUG { 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:09:01 DEBUG "result": [ 05:09:01 DEBUG { 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:09:01 DEBUG }, 05:09:01 DEBUG { 05:09:01 DEBUG --- http response --- 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG 05:09:01 DEBUG "metric": { 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG "value": [ 05:09:01 DEBUG "metric": { 05:09:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:09:01 DEBUG { 05:09:01 DEBUG "data": { 05:09:01 DEBUG { 05:09:01 DEBUG "instance": "10.142.0.51:9100" 05:09:01 DEBUG "instance": "10.142.0.11:9100" 05:09:01 DEBUG { 05:09:01 DEBUG 05:09:01 DEBUG "instance": "10.142.0.11:9100" 05:09:01 DEBUG ----- output ----- 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG }, 05:09:01 DEBUG }, 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 05:09:01 DEBUG }, 05:09:01 DEBUG { 05:09:01 DEBUG "data": { 05:09:01 DEBUG "result": [ 05:09:01 DEBUG "data": { 05:09:01 DEBUG "value": [ 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG "value": [ 05:09:01 DEBUG "status": "success", 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG { 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG "data": { 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG "data": { 05:09:01 DEBUG "result": [ 05:09:01 DEBUG "metric": { 05:09:01 DEBUG "result": [ 05:09:01 DEBUG "0.0009999999999995644" 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG [...] 05:09:01 DEBUG [...] 05:09:01 DEBUG 05:09:01 DEBUG "resultType": "vector", 05:09:01 DEBUG { 05:09:01 DEBUG "pod": "ds-idrepo-0" 05:09:01 DEBUG { 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG "result": [ 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG "result": [ 05:09:01 DEBUG "metric": { 05:09:01 DEBUG }, 05:09:01 DEBUG "metric": { 05:09:01 DEBUG 05:09:01 DEBUG ] 05:09:01 DEBUG { 05:09:01 DEBUG 05:09:01 DEBUG [...] 05:09:01 DEBUG { 05:09:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:09:01 DEBUG "value": [ 05:09:01 DEBUG "pod": "ds-idrepo-0" 05:09:01 DEBUG 05:09:01 DEBUG } 05:09:01 DEBUG "metric": { 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG "metric": { 05:09:01 DEBUG }, 05:09:01 DEBUG 05:09:01 DEBUG }, 05:09:01 DEBUG 05:09:01 DEBUG } 05:09:01 DEBUG "pod": "ds-idrepo-0" 05:09:01 DEBUG "instance": "10.142.0.51:9100" 05:09:01 DEBUG 05:09:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 05:09:01 DEBUG "value": [ 05:09:01 DEBUG "instance": "10.142.0.51:9100" 05:09:01 DEBUG }, 05:09:01 DEBUG }, 05:09:01 DEBUG 05:09:01 DEBUG }, 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG }, 05:09:01 DEBUG "value": [ 05:09:01 DEBUG "value": [ 05:09:01 DEBUG "instance": "10.142.0.51:9100" 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 05:09:01 DEBUG [...] 05:09:01 DEBUG 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG }, 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG 05:09:01 DEBUG "2398028.2666666675" 05:09:01 DEBUG 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG [...] 05:09:01 DEBUG "2381227.466666667" 05:09:01 DEBUG 05:09:01 DEBUG ] 05:09:01 DEBUG [...] 05:09:01 DEBUG "19797.333333333336" 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG 05:09:01 DEBUG ] 05:09:01 DEBUG [...] 05:09:01 DEBUG } 05:09:01 DEBUG 05:09:01 DEBUG ] 05:09:01 DEBUG [...] 05:09:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:09:01 DEBUG 05:09:01 DEBUG } 05:09:01 DEBUG 05:09:01 DEBUG ] 05:09:01 DEBUG 05:09:01 DEBUG } 05:09:01 DEBUG 05:09:01 DEBUG }, 05:09:01 DEBUG 05:09:01 DEBUG ] 05:09:01 DEBUG 05:09:01 DEBUG } 05:09:01 DEBUG 05:09:01 DEBUG ] 05:09:01 DEBUG 05:09:01 DEBUG "value": [ 05:09:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:09:01 DEBUG } 05:09:01 DEBUG 05:09:01 DEBUG } 05:09:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:09:01 DEBUG } 05:09:01 DEBUG 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG }, 05:09:01 DEBUG } 05:09:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:09:01 DEBUG }, 05:09:01 DEBUG } 05:09:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:09:01 DEBUG "406.48362553754544" 05:09:01 DEBUG "value": [ 05:09:01 DEBUG }, 05:09:01 DEBUG "value": [ 05:09:01 DEBUG }, 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG "value": [ 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG ] 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG "0" 05:09:01 DEBUG 1735362541, 05:09:01 DEBUG "0.001181966440664505" 05:09:01 DEBUG } 05:09:01 DEBUG "0.17810373789910627" 05:09:01 DEBUG ] 05:09:01 DEBUG "14875963392" 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG } 05:09:01 DEBUG } 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG ] 05:09:01 DEBUG } 05:09:01 DEBUG } 05:09:01 DEBUG } 05:09:01 DEBUG } 05:09:01 DEBUG } 05:09:31 INFO 05:09:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:09:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:09:31 INFO [loop_until]: OK (rc = 0) 05:09:31 DEBUG --- stdout --- 05:09:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 446m 13375Mi rcs-687d95c5b-qs2xv 102m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:09:31 DEBUG --- stderr --- 05:09:31 DEBUG 05:09:33 INFO 05:09:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:09:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:09:33 INFO [loop_until]: OK (rc = 0) 05:09:33 DEBUG --- stdout --- 05:09:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 214m 1% 3422Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 637m 2% 18877Mi 66% 05:09:33 DEBUG --- stderr --- 05:09:33 DEBUG 05:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362600" 05:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362600" 05:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362600" 05:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362600" 05:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362600" 05:10:00 INFO [http_cmd]: http status code OK 05:10:00 DEBUG --- status code --- 05:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:00 DEBUG --- http response --- 05:10:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:00 INFO [http_cmd]: http status code OK 05:10:00 DEBUG --- status code --- 05:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:00 DEBUG --- http response --- 05:10:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:00 INFO [http_cmd]: http status code OK 05:10:00 DEBUG --- status code --- 05:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:00 DEBUG --- http response --- 05:10:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:00 INFO [http_cmd]: http status code OK 05:10:00 DEBUG --- status code --- 05:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:00 DEBUG --- http response --- 05:10:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:00 INFO [http_cmd]: http status code OK 05:10:00 DEBUG --- status code --- 05:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:00 DEBUG --- http response --- 05:10:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 ERROR [http_cmd]: ERROR 05:10:01 ERROR --- status code --- 05:10:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:10:01 ERROR --- http response --- 05:10:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:10:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG { 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG "data": { 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG "result": [ 05:10:01 DEBUG { 05:10:01 DEBUG "metric": { 05:10:01 DEBUG "pod": "ds-idrepo-0" 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG [...] 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG "831016960" 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362601" 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG { 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG "data": { 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG "result": [ 05:10:01 DEBUG { 05:10:01 DEBUG "metric": { 05:10:01 DEBUG "instance": "10.142.0.11:9100" 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG [...] 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "instance": "10.142.0.51:9100" 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG "0.0005666666666722147" 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362601" 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362601" 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG { 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG "data": { 05:10:01 DEBUG "resultType": "vector", 05:10:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362601" 05:10:01 DEBUG "result": [ 05:10:01 DEBUG { 05:10:01 DEBUG "metric": { 05:10:01 DEBUG "instance": "10.142.0.11:9100" 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG [...] 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "instance": "10.142.0.51:9100" 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG }, 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG "value": [ 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG "0.0006666666666670078" 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG ] 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG } 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG { 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG ] 05:10:01 DEBUG { 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG } 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG "data": { 05:10:01 DEBUG { 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG } 05:10:01 DEBUG "data": { 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG "result": [ 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG { 05:10:01 DEBUG { 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:10:01 INFO [http_cmd]: http status code OK 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:10:01 DEBUG "result": [ 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:10:01 DEBUG "data": { 05:10:01 DEBUG "metric": { 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG { 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG { 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG "instance": "10.142.0.11:9100" 05:10:01 DEBUG "data": { 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG { 05:10:01 DEBUG "metric": { 05:10:01 DEBUG { 05:10:01 DEBUG "result": [ 05:10:01 DEBUG }, 05:10:01 DEBUG --- status code --- 05:10:01 DEBUG "result": [ 05:10:01 DEBUG "data": { 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG "instance": "10.142.0.11:9100" 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG { 05:10:01 DEBUG "value": [ 05:10:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:10:01 DEBUG { 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG "data": { 05:10:01 DEBUG }, 05:10:01 DEBUG "data": { 05:10:01 DEBUG "metric": { 05:10:01 DEBUG 05:10:01 DEBUG --- http response --- 05:10:01 DEBUG "metric": { 05:10:01 DEBUG "result": [ 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG "result": [ 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG "instance": "10.142.0.11:9100" 05:10:01 DEBUG 05:10:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:10:01 DEBUG "pod": "ds-idrepo-0" 05:10:01 DEBUG { 05:10:01 DEBUG "metric": { 05:10:01 DEBUG { 05:10:01 DEBUG "result": [ 05:10:01 DEBUG }, 05:10:01 DEBUG 05:10:01 DEBUG ----- output ----- 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:10:01 DEBUG "metric": { 05:10:01 DEBUG { 05:10:01 DEBUG "value": [ 05:10:01 DEBUG [...] 05:10:01 DEBUG { 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 05:10:01 DEBUG }, 05:10:01 DEBUG "pod": "ds-idrepo-0" 05:10:01 DEBUG "metric": { 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "status": "success", 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "value": [ 05:10:01 DEBUG }, 05:10:01 DEBUG "pod": "ds-idrepo-0" 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "data": { 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "value": [ 05:10:01 DEBUG }, 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "resultType": "vector", 05:10:01 DEBUG 05:10:01 DEBUG [...] 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "value": [ 05:10:01 DEBUG [...] 05:10:01 DEBUG "instance": "10.142.0.51:9100" 05:10:01 DEBUG "result": [ 05:10:01 DEBUG [...] 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG }, 05:10:01 DEBUG { 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG [...] 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "value": [ 05:10:01 DEBUG "metric": { 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG [...] 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:10:01 DEBUG 05:10:01 DEBUG "instance": "10.142.0.51:9100" 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG [...] 05:10:01 DEBUG 05:10:01 DEBUG "instance": "10.142.0.51:9100" 05:10:01 DEBUG }, 05:10:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:10:01 DEBUG }, 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "1236551.1333333333" 05:10:01 DEBUG 05:10:01 DEBUG 05:10:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:10:01 DEBUG 05:10:01 DEBUG ] 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG }, 05:10:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:10:01 DEBUG } 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG "697.4753538960185" 05:10:01 DEBUG "value": [ 05:10:01 DEBUG }, 05:10:01 DEBUG ] 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG "23210.66666666667" 05:10:01 DEBUG ] 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG } 05:10:01 DEBUG "1251606.8" 05:10:01 DEBUG 05:10:01 DEBUG "0" 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG "14880202752" 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG } 05:10:01 DEBUG ] 05:10:01 DEBUG [...] 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG ] 05:10:01 DEBUG ] 05:10:01 DEBUG "0.0011677813813760706" 05:10:01 DEBUG } 05:10:01 DEBUG 05:10:01 DEBUG } 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG ] 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 DEBUG ] 05:10:01 DEBUG ] 05:10:01 DEBUG 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 DEBUG 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:10:01 DEBUG }, 05:10:01 DEBUG "value": [ 05:10:01 DEBUG 1735362601, 05:10:01 DEBUG "0.33142015704868305" 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG ] 05:10:01 DEBUG } 05:10:01 DEBUG } 05:10:31 INFO 05:10:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:10:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:10:31 INFO [loop_until]: OK (rc = 0) 05:10:31 DEBUG --- stdout --- 05:10:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 63m 13375Mi rcs-687d95c5b-qs2xv 80m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:10:31 DEBUG --- stderr --- 05:10:31 DEBUG 05:10:33 INFO 05:10:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:10:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:10:33 INFO [loop_until]: OK (rc = 0) 05:10:33 DEBUG --- stdout --- 05:10:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 131m 0% 3436Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 86m 0% 18877Mi 66% 05:10:33 DEBUG --- stderr --- 05:10:33 DEBUG 05:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362660" 05:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362660" 05:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362660" 05:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362660" 05:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362660" 05:11:00 INFO [http_cmd]: http status code OK 05:11:00 DEBUG --- status code --- 05:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:00 DEBUG --- http response --- 05:11:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:00 INFO [http_cmd]: http status code OK 05:11:00 DEBUG --- status code --- 05:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:00 DEBUG --- http response --- 05:11:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:00 INFO [http_cmd]: http status code OK 05:11:00 DEBUG --- status code --- 05:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:00 DEBUG --- http response --- 05:11:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:00 INFO [http_cmd]: http status code OK 05:11:00 DEBUG --- status code --- 05:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:00 DEBUG --- http response --- 05:11:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:00 INFO [http_cmd]: http status code OK 05:11:00 DEBUG --- status code --- 05:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:00 DEBUG --- http response --- 05:11:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG --- status code --- 05:11:01 ERROR [http_cmd]: ERROR 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 ERROR --- status code --- 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG --- http response --- 05:11:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 ERROR --- http response --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:11:01 DEBUG --- http response --- 05:11:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG { 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG "data": { 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG "result": [ 05:11:01 DEBUG { 05:11:01 DEBUG "metric": { 05:11:01 DEBUG "pod": "ds-idrepo-0" 05:11:01 DEBUG }, 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG [...] 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:11:01 DEBUG }, 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG "831025152" 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG } 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362661" 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG { 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG "data": { 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG "result": [ 05:11:01 DEBUG { 05:11:01 DEBUG "metric": { 05:11:01 DEBUG "instance": "10.142.0.11:9100" 05:11:01 DEBUG }, 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG [...] 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG "instance": "10.142.0.51:9100" 05:11:01 DEBUG }, 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG "0.0005333333333320904" 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG } 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362661" 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG { 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG "data": { 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG "result": [ 05:11:01 DEBUG { 05:11:01 DEBUG "metric": { 05:11:01 DEBUG "instance": "10.142.0.11:9100" 05:11:01 DEBUG }, 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 DEBUG 05:11:01 DEBUG [...] 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG "instance": "10.142.0.51:9100" 05:11:01 DEBUG }, 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG "0" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362661" 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG } 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362661" 05:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362661" 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG { 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG "data": { 05:11:01 DEBUG { 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG "result": [ 05:11:01 DEBUG "data": { 05:11:01 DEBUG { 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG "metric": { 05:11:01 DEBUG "result": [ 05:11:01 DEBUG "instance": "10.142.0.11:9100" 05:11:01 DEBUG { 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG }, 05:11:01 DEBUG "metric": { 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG "value": [ 05:11:01 DEBUG "pod": "ds-idrepo-0" 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG 05:11:01 DEBUG }, 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG 05:11:01 DEBUG "value": [ 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG { 05:11:01 DEBUG [...] 05:11:01 DEBUG { 05:11:01 DEBUG 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG "data": { 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG 05:11:01 DEBUG [...] 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG "data": { 05:11:01 DEBUG "result": [ 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG { 05:11:01 DEBUG "instance": "10.142.0.51:9100" 05:11:01 DEBUG 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG "result": [ 05:11:01 DEBUG "metric": { 05:11:01 DEBUG }, 05:11:01 DEBUG 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:11:01 DEBUG { 05:11:01 DEBUG "pod": "ds-idrepo-0" 05:11:01 DEBUG "value": [ 05:11:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG "metric": { 05:11:01 DEBUG }, 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG }, 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG { 05:11:01 DEBUG "instance": "10.142.0.11:9100" 05:11:01 DEBUG "value": [ 05:11:01 DEBUG "80790.13333333336" 05:11:01 DEBUG "value": [ 05:11:01 DEBUG --- status code --- 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 INFO [http_cmd]: http status code OK 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG }, 05:11:01 DEBUG 05:11:01 DEBUG ] 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG --- status code --- 05:11:01 DEBUG "data": { 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 05:11:01 DEBUG } 05:11:01 DEBUG "700.5002351532771" 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG ] 05:11:01 DEBUG ] 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG --- http response --- 05:11:01 DEBUG "result": [ 05:11:01 DEBUG 05:11:01 DEBUG [...] 05:11:01 DEBUG } 05:11:01 DEBUG } 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:11:01 DEBUG { 05:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG } 05:11:01 DEBUG ] 05:11:01 DEBUG { 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG "metric": { 05:11:01 DEBUG "pod": "ds-idrepo-0" 05:11:01 DEBUG [...] 05:11:01 DEBUG 05:11:01 DEBUG } 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG { 05:11:01 DEBUG ----- output ----- 05:11:01 DEBUG }, 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG } 05:11:01 DEBUG "data": { 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG { 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 05:11:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG "data": { 05:11:01 DEBUG "status": "success", 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG }, 05:11:01 DEBUG "result": [ 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG "data": { 05:11:01 DEBUG 05:11:01 DEBUG "instance": "10.142.0.51:9100" 05:11:01 DEBUG "value": [ 05:11:01 DEBUG { 05:11:01 DEBUG "result": [ 05:11:01 DEBUG { 05:11:01 DEBUG 05:11:01 DEBUG }, 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG "metric": { 05:11:01 DEBUG "resultType": "vector", 05:11:01 DEBUG "metric": { 05:11:01 DEBUG [...] 05:11:01 DEBUG "value": [ 05:11:01 DEBUG "0.0011305213898032142" 05:11:01 DEBUG "instance": "10.142.0.11:9100" 05:11:01 DEBUG "result": [ 05:11:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:11:01 DEBUG 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG ] 05:11:01 DEBUG }, 05:11:01 DEBUG { 05:11:01 DEBUG }, 05:11:01 DEBUG 05:11:01 DEBUG "95943.00000000001" 05:11:01 DEBUG } 05:11:01 DEBUG "value": [ 05:11:01 DEBUG "metric": { 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 05:11:01 DEBUG ] 05:11:01 DEBUG ] 05:11:01 DEBUG 05:11:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:11:01 DEBUG 05:11:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:11:01 DEBUG } 05:11:01 DEBUG } 05:11:01 DEBUG 05:11:01 DEBUG }, 05:11:01 DEBUG 05:11:01 DEBUG }, 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG 05:11:01 DEBUG "value": [ 05:11:01 DEBUG 05:11:01 DEBUG "value": [ 05:11:01 DEBUG } 05:11:01 DEBUG [...] 05:11:01 DEBUG 05:11:01 DEBUG [...] 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG } 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG "0" 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG 05:11:01 DEBUG ] 05:11:01 DEBUG 05:11:01 DEBUG [...] 05:11:01 DEBUG 05:11:01 DEBUG } 05:11:01 DEBUG "instance": "10.142.0.51:9100" 05:11:01 DEBUG 05:11:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:11:01 DEBUG ] 05:11:01 DEBUG }, 05:11:01 DEBUG "value": [ 05:11:01 DEBUG }, 05:11:01 DEBUG } 05:11:01 DEBUG 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG "value": [ 05:11:01 DEBUG } 05:11:01 DEBUG 05:11:01 DEBUG "25941.333333333336" 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG "14879715328" 05:11:01 DEBUG ] 05:11:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG }, 05:11:01 DEBUG } 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG ] 05:11:01 DEBUG "value": [ 05:11:01 DEBUG } 05:11:01 DEBUG } 05:11:01 DEBUG 1735362661, 05:11:01 DEBUG } 05:11:01 DEBUG "0.036649057949669446" 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG ] 05:11:01 DEBUG } 05:11:01 DEBUG } 05:11:31 INFO 05:11:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:11:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:11:31 INFO [loop_until]: OK (rc = 0) 05:11:31 DEBUG --- stdout --- 05:11:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 533Mi ds-idrepo-0 6m 13377Mi rcs-687d95c5b-qs2xv 78m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:11:31 DEBUG --- stderr --- 05:11:31 DEBUG 05:11:33 INFO 05:11:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:11:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:11:33 INFO [loop_until]: OK (rc = 0) 05:11:33 DEBUG --- stdout --- 05:11:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 138m 0% 3450Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 128m 0% 18900Mi 66% 05:11:33 DEBUG --- stderr --- 05:11:33 DEBUG 05:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362720" 05:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362720" 05:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362720" 05:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362720" 05:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362720" 05:12:00 INFO [http_cmd]: http status code OK 05:12:00 DEBUG --- status code --- 05:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:00 DEBUG --- http response --- 05:12:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:00 INFO [http_cmd]: http status code OK 05:12:00 DEBUG --- status code --- 05:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:00 DEBUG --- http response --- 05:12:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:00 INFO [http_cmd]: http status code OK 05:12:00 DEBUG --- status code --- 05:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:00 DEBUG --- http response --- 05:12:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:00 INFO [http_cmd]: http status code OK 05:12:00 DEBUG --- status code --- 05:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:00 DEBUG --- http response --- 05:12:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:00 INFO [http_cmd]: http status code OK 05:12:00 DEBUG --- status code --- 05:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:00 DEBUG --- http response --- 05:12:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362721" 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 ERROR [http_cmd]: ERROR 05:12:01 ERROR --- status code --- 05:12:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:12:01 ERROR --- http response --- 05:12:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:12:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG { 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG "data": { 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG "result": [ 05:12:01 DEBUG { 05:12:01 DEBUG "metric": { 05:12:01 DEBUG "pod": "ds-idrepo-0" 05:12:01 DEBUG }, 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG [...] 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:12:01 DEBUG }, 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG "831025152" 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362721" 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG { 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG "data": { 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG "result": [ 05:12:01 DEBUG { 05:12:01 DEBUG "metric": { 05:12:01 DEBUG "instance": "10.142.0.11:9100" 05:12:01 DEBUG }, 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG [...] 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG "instance": "10.142.0.51:9100" 05:12:01 DEBUG }, 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG "0.001166666666673185" 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362721" 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG { 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG "data": { 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG "result": [ 05:12:01 DEBUG { 05:12:01 DEBUG "metric": { 05:12:01 DEBUG "instance": "10.142.0.11:9100" 05:12:01 DEBUG }, 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG [...] 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG "instance": "10.142.0.51:9100" 05:12:01 DEBUG }, 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG "0.0009999999999995644" 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362721" 05:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362721" 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG { 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG "data": { 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG "result": [ 05:12:01 DEBUG { 05:12:01 DEBUG "metric": { 05:12:01 DEBUG "instance": "10.142.0.11:9100" 05:12:01 DEBUG }, 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG [...] 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG "instance": "10.142.0.51:9100" 05:12:01 DEBUG }, 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG "1233577.1666666667" 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- http response --- 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG "status": "success", 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG "data": { 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG { 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG { 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG "result": [ 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG "data": { 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:12:01 DEBUG { 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 INFO [http_cmd]: http status code OK 05:12:01 DEBUG "data": { 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG "metric": { 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG --- status code --- 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG "result": [ 05:12:01 DEBUG { 05:12:01 DEBUG "pod": "ds-idrepo-0" 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:01 DEBUG "result": [ 05:12:01 DEBUG { 05:12:01 DEBUG { 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG }, 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG --- http response --- 05:12:01 DEBUG { 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG "metric": { 05:12:01 DEBUG "data": { 05:12:01 DEBUG "value": [ 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:12:01 DEBUG "metric": { 05:12:01 DEBUG "data": { 05:12:01 DEBUG "instance": "10.142.0.11:9100" 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG ----- output ----- 05:12:01 DEBUG "pod": "ds-idrepo-0" 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG }, 05:12:01 DEBUG "result": [ 05:12:01 DEBUG 05:12:01 DEBUG { 05:12:01 DEBUG { 05:12:01 DEBUG }, 05:12:01 DEBUG "result": [ 05:12:01 DEBUG "value": [ 05:12:01 DEBUG { 05:12:01 DEBUG 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG "status": "success", 05:12:01 DEBUG "value": [ 05:12:01 DEBUG { 05:12:01 DEBUG 05:12:01 DEBUG "metric": { 05:12:01 DEBUG [...] 05:12:01 DEBUG "data": { 05:12:01 DEBUG "data": { 05:12:01 DEBUG 05:12:01 DEBUG "metric": { 05:12:01 DEBUG 05:12:01 DEBUG "pod": "ds-idrepo-0" 05:12:01 DEBUG 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG "resultType": "vector", 05:12:01 DEBUG 05:12:01 DEBUG "instance": "10.142.0.11:9100" 05:12:01 DEBUG 05:12:01 DEBUG }, 05:12:01 DEBUG 05:12:01 DEBUG "result": [ 05:12:01 DEBUG "result": [ 05:12:01 DEBUG 05:12:01 DEBUG }, 05:12:01 DEBUG [...] 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 05:12:01 DEBUG { 05:12:01 DEBUG { 05:12:01 DEBUG [...] 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:12:01 DEBUG "metric": { 05:12:01 DEBUG "metric": { 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG }, 05:12:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:12:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG "value": [ 05:12:01 DEBUG }, 05:12:01 DEBUG }, 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG "instance": "10.142.0.51:9100" 05:12:01 DEBUG [...] 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG "value": [ 05:12:01 DEBUG "value": [ 05:12:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:12:01 DEBUG [...] 05:12:01 DEBUG }, 05:12:01 DEBUG 05:12:01 DEBUG "618.9887155455103" 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG }, 05:12:01 DEBUG 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 05:12:01 DEBUG ] 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG "value": [ 05:12:01 DEBUG 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG 05:12:01 DEBUG } 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG 05:12:01 DEBUG "1250529.4000000004" 05:12:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:12:01 DEBUG ] 05:12:01 DEBUG [...] 05:12:01 DEBUG [...] 05:12:01 DEBUG "0.0012049113435000332" 05:12:01 DEBUG "instance": "10.142.0.51:9100" 05:12:01 DEBUG ] 05:12:01 DEBUG }, 05:12:01 DEBUG } 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG ] 05:12:01 DEBUG }, 05:12:01 DEBUG } 05:12:01 DEBUG "value": [ 05:12:01 DEBUG } 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG } 05:12:01 DEBUG "value": [ 05:12:01 DEBUG ] 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG 05:12:01 DEBUG 05:12:01 DEBUG ] 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG } 05:12:01 DEBUG "0" 05:12:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:12:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:12:01 DEBUG } 05:12:01 DEBUG "24849.066666666673" 05:12:01 DEBUG } 05:12:01 DEBUG ] 05:12:01 DEBUG }, 05:12:01 DEBUG }, 05:12:01 DEBUG } 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG "value": [ 05:12:01 DEBUG "value": [ 05:12:01 DEBUG } 05:12:01 DEBUG ] 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG 1735362721, 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG "0.08656109987565336" 05:12:01 DEBUG "14902706176" 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:01 DEBUG ] 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:01 DEBUG ] 05:12:01 DEBUG ] 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:01 DEBUG } 05:12:31 INFO 05:12:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:12:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:12:31 INFO [loop_until]: OK (rc = 0) 05:12:31 DEBUG --- stdout --- 05:12:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 62m 13376Mi rcs-687d95c5b-qs2xv 64m 425Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:12:31 DEBUG --- stderr --- 05:12:31 DEBUG 05:12:33 INFO 05:12:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:12:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:12:33 INFO [loop_until]: OK (rc = 0) 05:12:33 DEBUG --- stdout --- 05:12:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 176m 1% 3448Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 133m 0% 18901Mi 66% 05:12:33 DEBUG --- stderr --- 05:12:33 DEBUG 05:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362780" 05:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362780" 05:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362780" 05:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362780" 05:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362780" 05:13:00 INFO [http_cmd]: http status code OK 05:13:00 DEBUG --- status code --- 05:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:00 DEBUG --- http response --- 05:13:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:00 INFO [http_cmd]: http status code OK 05:13:00 DEBUG --- status code --- 05:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:00 DEBUG --- http response --- 05:13:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:00 INFO [http_cmd]: http status code OK 05:13:00 DEBUG --- status code --- 05:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:00 DEBUG --- http response --- 05:13:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:00 INFO [http_cmd]: http status code OK 05:13:00 DEBUG --- status code --- 05:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:00 DEBUG --- http response --- 05:13:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:00 INFO [http_cmd]: http status code OK 05:13:00 DEBUG --- status code --- 05:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:00 DEBUG --- http response --- 05:13:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 ERROR [http_cmd]: ERROR 05:13:01 ERROR --- status code --- 05:13:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:13:01 ERROR --- http response --- 05:13:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:13:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG { 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG "data": { 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG "result": [ 05:13:01 DEBUG { 05:13:01 DEBUG "metric": { 05:13:01 DEBUG "pod": "ds-idrepo-0" 05:13:01 DEBUG }, 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG [...] 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:13:01 DEBUG }, 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG "831033344" 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362781" 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG { 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG "data": { 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG "result": [ 05:13:01 DEBUG { 05:13:01 DEBUG "metric": { 05:13:01 DEBUG "instance": "10.142.0.11:9100" 05:13:01 DEBUG }, 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG [...] 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "instance": "10.142.0.51:9100" 05:13:01 DEBUG }, 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG "0.0005333333333396695" 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362781" 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG { 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG "data": { 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG "result": [ 05:13:01 DEBUG { 05:13:01 DEBUG "metric": { 05:13:01 DEBUG "instance": "10.142.0.11:9100" 05:13:01 DEBUG }, 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG [...] 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "instance": "10.142.0.51:9100" 05:13:01 DEBUG }, 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG "0.0003333333333335039" 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362781" 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG { 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG "data": { 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG "result": [ 05:13:01 DEBUG { 05:13:01 DEBUG "metric": { 05:13:01 DEBUG "instance": "10.142.0.11:9100" 05:13:01 DEBUG }, 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG [...] 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "instance": "10.142.0.51:9100" 05:13:01 DEBUG }, 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG "2370552.033333334" 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362781" 05:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362781" 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG { 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG "data": { 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG { 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG "result": [ 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:13:01 DEBUG "data": { 05:13:01 DEBUG { 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG "metric": { 05:13:01 DEBUG { 05:13:01 DEBUG "result": [ 05:13:01 DEBUG "instance": "10.142.0.11:9100" 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG { 05:13:01 DEBUG }, 05:13:01 DEBUG "data": { 05:13:01 DEBUG "metric": { 05:13:01 DEBUG "value": [ 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG "pod": "ds-idrepo-0" 05:13:01 DEBUG 05:13:01 DEBUG "result": [ 05:13:01 DEBUG }, 05:13:01 DEBUG 05:13:01 DEBUG { 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 05:13:01 DEBUG "metric": { 05:13:01 DEBUG 05:13:01 DEBUG [...] 05:13:01 DEBUG "pod": "ds-idrepo-0" 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG }, 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG "value": [ 05:13:01 DEBUG [...] 05:13:01 DEBUG 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "instance": "10.142.0.51:9100" 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG }, 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "value": [ 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG [...] 05:13:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG { 05:13:01 DEBUG { 05:13:01 DEBUG 05:13:01 DEBUG }, 05:13:01 DEBUG "2387219.766666667" 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG 05:13:01 DEBUG "value": [ 05:13:01 DEBUG ] 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 INFO [http_cmd]: http status code OK 05:13:01 DEBUG "data": { 05:13:01 DEBUG "data": { 05:13:01 DEBUG 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG } 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG --- status code --- 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:13:01 DEBUG "0.0011939600349704727" 05:13:01 DEBUG ] 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:01 DEBUG "result": [ 05:13:01 DEBUG "result": [ 05:13:01 DEBUG }, 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG --- http response --- 05:13:01 DEBUG { 05:13:01 DEBUG { 05:13:01 DEBUG "value": [ 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG "metric": { 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG ] 05:13:01 DEBUG ----- output ----- 05:13:01 DEBUG "metric": { 05:13:01 DEBUG { 05:13:01 DEBUG "pod": "ds-idrepo-0" 05:13:01 DEBUG "1077.0810943476763" 05:13:01 DEBUG } 05:13:01 DEBUG { 05:13:01 DEBUG "instance": "10.142.0.11:9100" 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG }, 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG "status": "success", 05:13:01 DEBUG }, 05:13:01 DEBUG "data": { 05:13:01 DEBUG "value": [ 05:13:01 DEBUG } 05:13:01 DEBUG "data": { 05:13:01 DEBUG "value": [ 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG 05:13:01 DEBUG ] 05:13:01 DEBUG "resultType": "vector", 05:13:01 DEBUG 05:13:01 DEBUG "result": [ 05:13:01 DEBUG 05:13:01 DEBUG } 05:13:01 DEBUG "result": [ 05:13:01 DEBUG { 05:13:01 DEBUG { 05:13:01 DEBUG 05:13:01 DEBUG } 05:13:01 DEBUG 05:13:01 DEBUG "metric": { 05:13:01 DEBUG "metric": { 05:13:01 DEBUG [...] 05:13:01 DEBUG 05:13:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:13:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:13:01 DEBUG 05:13:01 DEBUG [...] 05:13:01 DEBUG }, 05:13:01 DEBUG }, 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "value": [ 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG }, 05:13:01 DEBUG "instance": "10.142.0.51:9100" 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "value": [ 05:13:01 DEBUG }, 05:13:01 DEBUG [...] 05:13:01 DEBUG [...] 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG "value": [ 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG "0" 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG ] 05:13:01 DEBUG "20889.600000000002" 05:13:01 DEBUG 05:13:01 DEBUG 05:13:01 DEBUG } 05:13:01 DEBUG ] 05:13:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:13:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG }, 05:13:01 DEBUG }, 05:13:01 DEBUG } 05:13:01 DEBUG ] 05:13:01 DEBUG "value": [ 05:13:01 DEBUG "value": [ 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG 1735362781, 05:13:01 DEBUG } 05:13:01 DEBUG "14907469824" 05:13:01 DEBUG "0.08663127999198635" 05:13:01 DEBUG ] 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:01 DEBUG ] 05:13:01 DEBUG ] 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:01 DEBUG } 05:13:31 INFO 05:13:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:13:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:13:31 INFO [loop_until]: OK (rc = 0) 05:13:31 DEBUG --- stdout --- 05:13:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 73m 13376Mi rcs-687d95c5b-qs2xv 112m 425Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:13:31 DEBUG --- stderr --- 05:13:31 DEBUG 05:13:33 INFO 05:13:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:13:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:13:33 INFO [loop_until]: OK (rc = 0) 05:13:33 DEBUG --- stdout --- 05:13:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 207m 1% 3452Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 139m 0% 18880Mi 66% 05:13:33 DEBUG --- stderr --- 05:13:33 DEBUG 05:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362840" 05:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362840" 05:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362840" 05:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362840" 05:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362840" 05:14:00 INFO [http_cmd]: http status code OK 05:14:00 DEBUG --- status code --- 05:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:00 DEBUG --- http response --- 05:14:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:00 INFO [http_cmd]: http status code OK 05:14:00 INFO [http_cmd]: http status code OK 05:14:00 DEBUG --- status code --- 05:14:00 DEBUG --- status code --- 05:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:00 DEBUG --- http response --- 05:14:00 DEBUG --- http response --- 05:14:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:00 INFO [http_cmd]: http status code OK 05:14:00 DEBUG --- status code --- 05:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:00 DEBUG --- http response --- 05:14:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:00 INFO [http_cmd]: http status code OK 05:14:00 DEBUG --- status code --- 05:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:00 DEBUG --- http response --- 05:14:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 ERROR [http_cmd]: ERROR 05:14:01 ERROR --- status code --- 05:14:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 ERROR --- http response --- 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG --- status code --- 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- status code --- 05:14:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG --- http response --- 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG --- http response --- 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG { 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG "data": { 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG "result": [ 05:14:01 DEBUG { 05:14:01 DEBUG "metric": { 05:14:01 DEBUG "pod": "ds-idrepo-0" 05:14:01 DEBUG }, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG [...] 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:14:01 DEBUG }, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG "831033344" 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362841" 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG { 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG "data": { 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG "result": [ 05:14:01 DEBUG { 05:14:01 DEBUG "metric": { 05:14:01 DEBUG "instance": "10.142.0.11:9100" 05:14:01 DEBUG }, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG [...] 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG "instance": "10.142.0.51:9100" 05:14:01 DEBUG }, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG "0.0006333333333297258" 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362841" 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG { 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG "data": { 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG "result": [ 05:14:01 DEBUG { 05:14:01 DEBUG "metric": { 05:14:01 DEBUG "instance": "10.142.0.11:9100" 05:14:01 DEBUG }, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG [...] 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG "instance": "10.142.0.51:9100" 05:14:01 DEBUG }, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG "0" 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362841" 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG { 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG "data": { 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG "result": [ 05:14:01 DEBUG { 05:14:01 DEBUG "metric": { 05:14:01 DEBUG "instance": "10.142.0.11:9100" 05:14:01 DEBUG }, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG [...] 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG "instance": "10.142.0.51:9100" 05:14:01 DEBUG }, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG "1232704.9333333336" 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362841" 05:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362841" 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG { 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG "data": { 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG { 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG { 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG "result": [ 05:14:01 DEBUG { 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG "data": { 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG "metric": { 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG "data": { 05:14:01 DEBUG "pod": "ds-idrepo-0" 05:14:01 DEBUG "result": [ 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG }, 05:14:01 DEBUG { 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG "result": [ 05:14:01 DEBUG "value": [ 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG "metric": { 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 INFO [http_cmd]: http status code OK 05:14:01 DEBUG { 05:14:01 DEBUG { 05:14:01 DEBUG 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG "pod": "ds-idrepo-0" 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG --- status code --- 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG "metric": { 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG }, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 05:14:01 DEBUG "data": { 05:14:01 DEBUG "instance": "10.142.0.11:9100" 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG [...] 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:01 DEBUG 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG }, 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG --- http response --- 05:14:01 DEBUG 05:14:01 DEBUG "result": [ 05:14:01 DEBUG "value": [ 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:14:01 DEBUG 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:14:01 DEBUG [...] 05:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:14:01 DEBUG { 05:14:01 DEBUG 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG 05:14:01 DEBUG ----- output ----- 05:14:01 DEBUG "metric": { 05:14:01 DEBUG 05:14:01 DEBUG { 05:14:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:14:01 DEBUG { 05:14:01 DEBUG 05:14:01 DEBUG { 05:14:01 DEBUG "pod": "ds-idrepo-0" 05:14:01 DEBUG 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG }, 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG 05:14:01 DEBUG "status": "success", 05:14:01 DEBUG }, 05:14:01 DEBUG [...] 05:14:01 DEBUG "data": { 05:14:01 DEBUG "value": [ 05:14:01 DEBUG "data": { 05:14:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:14:01 DEBUG "data": { 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG }, 05:14:01 DEBUG "resultType": "vector", 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG "result": [ 05:14:01 DEBUG "0.0010381871330166232" 05:14:01 DEBUG "result": [ 05:14:01 DEBUG "value": [ 05:14:01 DEBUG "result": [ 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG { 05:14:01 DEBUG ] 05:14:01 DEBUG { 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG { 05:14:01 DEBUG 05:14:01 DEBUG "instance": "10.142.0.51:9100" 05:14:01 DEBUG "metric": { 05:14:01 DEBUG } 05:14:01 DEBUG "metric": { 05:14:01 DEBUG "993.5177962195473" 05:14:01 DEBUG "metric": { 05:14:01 DEBUG [...] 05:14:01 DEBUG }, 05:14:01 DEBUG "instance": "10.142.0.11:9100" 05:14:01 DEBUG ] 05:14:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:14:01 DEBUG ] 05:14:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:14:01 DEBUG 05:14:01 DEBUG "value": [ 05:14:01 DEBUG }, 05:14:01 DEBUG } 05:14:01 DEBUG }, 05:14:01 DEBUG } 05:14:01 DEBUG }, 05:14:01 DEBUG 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG "value": [ 05:14:01 DEBUG } 05:14:01 DEBUG "value": [ 05:14:01 DEBUG ] 05:14:01 DEBUG "value": [ 05:14:01 DEBUG 05:14:01 DEBUG "1249326.2666666668" 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG } 05:14:01 DEBUG 05:14:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:14:01 DEBUG ] 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG } 05:14:01 DEBUG 05:14:01 DEBUG }, 05:14:01 DEBUG } 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG "value": [ 05:14:01 DEBUG ] 05:14:01 DEBUG [...] 05:14:01 DEBUG [...] 05:14:01 DEBUG [...] 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG } 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG "0" 05:14:01 DEBUG } 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG ] 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG 05:14:01 DEBUG } 05:14:01 DEBUG "instance": "10.142.0.51:9100" 05:14:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:14:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:14:01 DEBUG ] 05:14:01 DEBUG }, 05:14:01 DEBUG }, 05:14:01 DEBUG }, 05:14:01 DEBUG } 05:14:01 DEBUG "value": [ 05:14:01 DEBUG "value": [ 05:14:01 DEBUG "value": [ 05:14:01 DEBUG } 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG 1735362841, 05:14:01 DEBUG "26077.86666666667" 05:14:01 DEBUG "0.09831818243809601" 05:14:01 DEBUG "14888849408" 05:14:01 DEBUG ] 05:14:01 DEBUG ] 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 DEBUG ] 05:14:01 DEBUG ] 05:14:01 DEBUG ] 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:01 DEBUG } 05:14:31 INFO 05:14:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:14:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:14:31 INFO [loop_until]: OK (rc = 0) 05:14:31 DEBUG --- stdout --- 05:14:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 50m 13375Mi rcs-687d95c5b-qs2xv 79m 425Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:14:31 DEBUG --- stderr --- 05:14:31 DEBUG 05:14:33 INFO 05:14:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:14:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:14:33 INFO [loop_until]: OK (rc = 0) 05:14:33 DEBUG --- stdout --- 05:14:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 134m 0% 3451Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 84m 0% 18880Mi 66% 05:14:33 DEBUG --- stderr --- 05:14:33 DEBUG 05:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362900" 05:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362900" 05:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362900" 05:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362900" 05:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362900" 05:15:00 INFO [http_cmd]: http status code OK 05:15:00 DEBUG --- status code --- 05:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:00 INFO [http_cmd]: http status code OK 05:15:00 INFO [http_cmd]: http status code OK 05:15:00 INFO [http_cmd]: http status code OK 05:15:00 INFO [http_cmd]: http status code OK 05:15:00 DEBUG --- http response --- 05:15:00 DEBUG --- status code --- 05:15:00 DEBUG --- status code --- 05:15:00 DEBUG --- status code --- 05:15:00 DEBUG --- status code --- 05:15:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:00 DEBUG --- http response --- 05:15:00 DEBUG --- http response --- 05:15:00 DEBUG --- http response --- 05:15:00 DEBUG --- http response --- 05:15:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 ERROR [http_cmd]: ERROR 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 ERROR --- status code --- 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG --- status code --- 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- status code --- 05:15:01 ERROR --- http response --- 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG --- http response --- 05:15:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG { 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG "data": { 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG "result": [ 05:15:01 DEBUG { 05:15:01 DEBUG "metric": { 05:15:01 DEBUG "pod": "ds-idrepo-0" 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG [...] 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG "831033344" 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG } 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362901" 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG { 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG "data": { 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG "result": [ 05:15:01 DEBUG { 05:15:01 DEBUG "metric": { 05:15:01 DEBUG "instance": "10.142.0.11:9100" 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG [...] 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG "instance": "10.142.0.51:9100" 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG "0.00036666666666936487" 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG } 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362901" 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG { 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG "data": { 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG "result": [ 05:15:01 DEBUG { 05:15:01 DEBUG "metric": { 05:15:01 DEBUG "instance": "10.142.0.11:9100" 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG [...] 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG "instance": "10.142.0.51:9100" 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG "0.0003333333333339776" 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG } 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362901" 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG { 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG "data": { 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG "result": [ 05:15:01 DEBUG { 05:15:01 DEBUG "metric": { 05:15:01 DEBUG "instance": "10.142.0.11:9100" 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG [...] 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG "instance": "10.142.0.51:9100" 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG "1238256.0333333334" 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG } 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362901" 05:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362901" 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG { 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG { 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG { 05:15:01 DEBUG { 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG "data": { 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG "data": { 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG "data": { 05:15:01 DEBUG "data": { 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG "result": [ 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG "resultType": "vector", 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 INFO [http_cmd]: http status code OK 05:15:01 DEBUG "result": [ 05:15:01 DEBUG { 05:15:01 DEBUG "result": [ 05:15:01 DEBUG "result": [ 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG --- status code --- 05:15:01 DEBUG { 05:15:01 DEBUG "metric": { 05:15:01 DEBUG { 05:15:01 DEBUG { 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:01 DEBUG "metric": { 05:15:01 DEBUG "pod": "ds-idrepo-0" 05:15:01 DEBUG "metric": { 05:15:01 DEBUG "metric": { 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG --- http response --- 05:15:01 DEBUG "pod": "ds-idrepo-0" 05:15:01 DEBUG }, 05:15:01 DEBUG "pod": "ds-idrepo-0" 05:15:01 DEBUG "instance": "10.142.0.11:9100" 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG }, 05:15:01 DEBUG }, 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG ----- output ----- 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 05:15:01 DEBUG "value": [ 05:15:01 DEBUG "value": [ 05:15:01 DEBUG { 05:15:01 DEBUG { 05:15:01 DEBUG { 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG "status": "success", 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG "data": { 05:15:01 DEBUG "data": { 05:15:01 DEBUG "data": { 05:15:01 DEBUG 05:15:01 DEBUG [...] 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG "resultType": "vector", 05:15:01 DEBUG [...] 05:15:01 DEBUG 05:15:01 DEBUG [...] 05:15:01 DEBUG [...] 05:15:01 DEBUG "result": [ 05:15:01 DEBUG "result": [ 05:15:01 DEBUG "result": [ 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG { 05:15:01 DEBUG { 05:15:01 DEBUG { 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG "metric": { 05:15:01 DEBUG "metric": { 05:15:01 DEBUG "metric": { 05:15:01 DEBUG 05:15:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:15:01 DEBUG 05:15:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:15:01 DEBUG "instance": "10.142.0.11:9100" 05:15:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:15:01 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:15:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:15:01 DEBUG }, 05:15:01 DEBUG 05:15:01 DEBUG }, 05:15:01 DEBUG }, 05:15:01 DEBUG }, 05:15:01 DEBUG }, 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG "instance": "10.142.0.51:9100" 05:15:01 DEBUG "value": [ 05:15:01 DEBUG "value": [ 05:15:01 DEBUG "value": [ 05:15:01 DEBUG "value": [ 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG }, 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG "909.7938805018825" 05:15:01 DEBUG "value": [ 05:15:01 DEBUG "0.0012175346357711585" 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG "0" 05:15:01 DEBUG ] 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG ] 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG "1253250.0000000002" 05:15:01 DEBUG } 05:15:01 DEBUG [...] 05:15:01 DEBUG [...] 05:15:01 DEBUG [...] 05:15:01 DEBUG } 05:15:01 DEBUG ] 05:15:01 DEBUG ] 05:15:01 DEBUG ] 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG } 05:15:01 DEBUG } 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG } 05:15:01 DEBUG } 05:15:01 DEBUG ] 05:15:01 DEBUG } 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG 05:15:01 DEBUG } 05:15:01 DEBUG } 05:15:01 DEBUG "instance": "10.142.0.51:9100" 05:15:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:15:01 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:15:01 DEBUG } 05:15:01 DEBUG }, 05:15:01 DEBUG }, 05:15:01 DEBUG }, 05:15:01 DEBUG "value": [ 05:15:01 DEBUG "value": [ 05:15:01 DEBUG "value": [ 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG 1735362901, 05:15:01 DEBUG "26077.86666666667" 05:15:01 DEBUG "0.05709473943760383" 05:15:01 DEBUG "14891057152" 05:15:01 DEBUG ] 05:15:01 DEBUG ] 05:15:01 DEBUG ] 05:15:02 DEBUG } 05:15:02 DEBUG } 05:15:02 DEBUG } 05:15:02 DEBUG ] 05:15:02 DEBUG ] 05:15:02 DEBUG ] 05:15:02 DEBUG } 05:15:02 DEBUG } 05:15:02 DEBUG } 05:15:02 DEBUG } 05:15:02 DEBUG } 05:15:02 DEBUG } 05:15:31 INFO 05:15:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:15:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:15:31 INFO [loop_until]: OK (rc = 0) 05:15:31 DEBUG --- stdout --- 05:15:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 50m 13376Mi rcs-687d95c5b-qs2xv 50m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:15:31 DEBUG --- stderr --- 05:15:31 DEBUG 05:15:33 INFO 05:15:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:15:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:15:33 INFO [loop_until]: OK (rc = 0) 05:15:33 DEBUG --- stdout --- 05:15:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 132m 0% 3447Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 95m 0% 18873Mi 66% 05:15:33 DEBUG --- stderr --- 05:15:33 DEBUG 05:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362960" 05:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362960" 05:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362960" 05:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735362960" 05:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362960" 05:16:00 INFO [http_cmd]: http status code OK 05:16:00 INFO [http_cmd]: http status code OK 05:16:00 DEBUG --- status code --- 05:16:00 INFO [http_cmd]: http status code OK 05:16:00 DEBUG --- status code --- 05:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:00 DEBUG --- status code --- 05:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:00 DEBUG --- http response --- 05:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:00 DEBUG --- http response --- 05:16:00 INFO [http_cmd]: http status code OK 05:16:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:00 DEBUG --- http response --- 05:16:00 INFO [http_cmd]: http status code OK 05:16:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:00 DEBUG --- status code --- 05:16:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:00 DEBUG --- status code --- 05:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:00 DEBUG --- http response --- 05:16:00 DEBUG --- http response --- 05:16:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG --- status code --- 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 ERROR [http_cmd]: ERROR 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 ERROR --- status code --- 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 ERROR --- http response --- 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:16:01 DEBUG ----- output ----- 05:16:01 DEBUG { 05:16:01 DEBUG "status": "success", 05:16:01 DEBUG "data": { 05:16:01 DEBUG "resultType": "vector", 05:16:01 DEBUG "result": [ 05:16:01 DEBUG { 05:16:01 DEBUG "metric": { 05:16:01 DEBUG "pod": "ds-idrepo-0" 05:16:01 DEBUG }, 05:16:01 DEBUG "value": [ 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG [...] 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:16:01 DEBUG }, 05:16:01 DEBUG "value": [ 05:16:01 DEBUG 1735362961, 05:16:01 DEBUG "831074304" 05:16:01 DEBUG ] 05:16:01 DEBUG } 05:16:01 DEBUG ] 05:16:01 DEBUG } 05:16:01 DEBUG } 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362961" 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:16:01 DEBUG ----- output ----- 05:16:01 DEBUG { 05:16:01 DEBUG "status": "success", 05:16:01 DEBUG "data": { 05:16:01 DEBUG "resultType": "vector", 05:16:01 DEBUG "result": [ 05:16:01 DEBUG { 05:16:01 DEBUG "metric": { 05:16:01 DEBUG "instance": "10.142.0.11:9100" 05:16:01 DEBUG }, 05:16:01 DEBUG "value": [ 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG [...] 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG "instance": "10.142.0.51:9100" 05:16:01 DEBUG }, 05:16:01 DEBUG "value": [ 05:16:01 DEBUG 1735362961, 05:16:01 DEBUG "0.0008666666666641733" 05:16:01 DEBUG ] 05:16:01 DEBUG } 05:16:01 DEBUG ] 05:16:01 DEBUG } 05:16:01 DEBUG } 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362961" 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:16:01 DEBUG ----- output ----- 05:16:01 DEBUG { 05:16:01 DEBUG "status": "success", 05:16:01 DEBUG "data": { 05:16:01 DEBUG "resultType": "vector", 05:16:01 DEBUG "result": [ 05:16:01 DEBUG { 05:16:01 DEBUG "metric": { 05:16:01 DEBUG "instance": "10.142.0.11:9100" 05:16:01 DEBUG }, 05:16:01 DEBUG "value": [ 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG [...] 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG "instance": "10.142.0.51:9100" 05:16:01 DEBUG }, 05:16:01 DEBUG "value": [ 05:16:01 DEBUG 1735362961, 05:16:01 DEBUG "0.0006666666666660605" 05:16:01 DEBUG ] 05:16:01 DEBUG } 05:16:01 DEBUG ] 05:16:01 DEBUG } 05:16:01 DEBUG } 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362961" 05:16:01 INFO [http_cmd]: http status code OK 05:16:01 DEBUG --- status code --- 05:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:01 DEBUG --- http response --- 05:16:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:16:01 DEBUG ----- output ----- 05:16:01 DEBUG { 05:16:01 DEBUG "status": "success", 05:16:01 DEBUG "data": { 05:16:01 DEBUG "resultType": "vector", 05:16:01 DEBUG "result": [ 05:16:01 DEBUG { 05:16:01 DEBUG "metric": { 05:16:01 DEBUG "instance": "10.142.0.11:9100" 05:16:01 DEBUG }, 05:16:01 DEBUG "value": [ 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG [...] 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG 05:16:01 DEBUG "instance": "10.142.0.51:9100" 05:16:01 DEBUG }, 05:16:01 DEBUG "value": [ 05:16:01 DEBUG 1735362961, 05:16:01 DEBUG "1204871.2666666668" 05:16:01 DEBUG ] 05:16:01 DEBUG } 05:16:01 DEBUG ] 05:16:01 DEBUG } 05:16:01 DEBUG } 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362961" 05:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735362962" 05:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362962" 05:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735362962" 05:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735362962" 05:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735362962" 05:16:02 INFO [http_cmd]: http status code OK 05:16:02 DEBUG --- status code --- 05:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:02 DEBUG --- http response --- 05:16:02 INFO [http_cmd]: http status code OK 05:16:02 INFO [http_cmd]: http status code OK 05:16:02 DEBUG --- status code --- 05:16:02 INFO [http_cmd]: http status code OK 05:16:02 INFO [http_cmd]: http status code OK 05:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:16:02 DEBUG --- status code --- 05:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:02 DEBUG --- status code --- 05:16:02 DEBUG --- status code --- 05:16:02 DEBUG ----- output ----- 05:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:02 DEBUG --- http response --- 05:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:02 DEBUG { 05:16:02 DEBUG --- http response --- 05:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:16:02 DEBUG --- http response --- 05:16:02 DEBUG --- http response --- 05:16:02 DEBUG "status": "success", 05:16:02 DEBUG ----- output ----- 05:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:16:02 DEBUG "data": { 05:16:02 DEBUG { 05:16:02 DEBUG ----- output ----- 05:16:02 DEBUG ----- output ----- 05:16:02 DEBUG ----- output ----- 05:16:02 DEBUG "resultType": "vector", 05:16:02 DEBUG "status": "success", 05:16:02 DEBUG { 05:16:02 DEBUG { 05:16:02 DEBUG { 05:16:02 DEBUG "result": [ 05:16:02 DEBUG "data": { 05:16:02 INFO [http_cmd]: http status code OK 05:16:02 INFO [http_cmd]: http status code OK 05:16:02 DEBUG "status": "success", 05:16:02 DEBUG "status": "success", 05:16:02 DEBUG "status": "success", 05:16:02 DEBUG { 05:16:02 DEBUG "resultType": "vector", 05:16:02 DEBUG --- status code --- 05:16:02 DEBUG --- status code --- 05:16:02 DEBUG "data": { 05:16:02 DEBUG "data": { 05:16:02 DEBUG "data": { 05:16:02 DEBUG "metric": { 05:16:02 DEBUG "result": [ 05:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:02 DEBUG "resultType": "vector", 05:16:02 DEBUG "resultType": "vector", 05:16:02 DEBUG "resultType": "vector", 05:16:02 DEBUG "pod": "ds-idrepo-0" 05:16:02 DEBUG { 05:16:02 DEBUG --- http response --- 05:16:02 DEBUG --- http response --- 05:16:02 DEBUG "result": [ 05:16:02 DEBUG "result": [ 05:16:02 DEBUG "result": [ 05:16:02 DEBUG }, 05:16:02 DEBUG "metric": { 05:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:16:02 DEBUG { 05:16:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:16:02 DEBUG { 05:16:02 DEBUG { 05:16:02 DEBUG "value": [ 05:16:02 DEBUG "instance": "10.142.0.11:9100" 05:16:02 DEBUG ----- output ----- 05:16:02 DEBUG "metric": { 05:16:02 DEBUG ----- output ----- 05:16:02 DEBUG "metric": { 05:16:02 DEBUG "metric": { 05:16:02 DEBUG 05:16:02 DEBUG }, 05:16:02 DEBUG { 05:16:02 DEBUG "pod": "ds-idrepo-0" 05:16:02 DEBUG { 05:16:02 DEBUG "pod": "ds-idrepo-0" 05:16:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:16:02 DEBUG 05:16:02 DEBUG "value": [ 05:16:02 DEBUG "status": "success", 05:16:02 DEBUG }, 05:16:02 DEBUG "status": "success", 05:16:02 DEBUG }, 05:16:02 DEBUG }, 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG "data": { 05:16:02 DEBUG "value": [ 05:16:02 DEBUG "data": { 05:16:02 DEBUG "value": [ 05:16:02 DEBUG "value": [ 05:16:02 DEBUG [...] 05:16:02 DEBUG 05:16:02 DEBUG "resultType": "vector", 05:16:02 DEBUG 05:16:02 DEBUG "resultType": "vector", 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG "result": [ 05:16:02 DEBUG 05:16:02 DEBUG "result": [ 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG [...] 05:16:02 DEBUG { 05:16:02 DEBUG 05:16:02 DEBUG { 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG "metric": { 05:16:02 DEBUG [...] 05:16:02 DEBUG "metric": { 05:16:02 DEBUG [...] 05:16:02 DEBUG [...] 05:16:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:16:02 DEBUG 05:16:02 DEBUG "instance": "10.142.0.11:9100" 05:16:02 DEBUG 05:16:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG }, 05:16:02 DEBUG 05:16:02 DEBUG }, 05:16:02 DEBUG 05:16:02 DEBUG }, 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG "value": [ 05:16:02 DEBUG "instance": "10.142.0.51:9100" 05:16:02 DEBUG "value": [ 05:16:02 DEBUG 05:16:02 DEBUG "value": [ 05:16:02 DEBUG 05:16:02 DEBUG 05:16:02 DEBUG 1735362962, 05:16:02 DEBUG }, 05:16:02 DEBUG 05:16:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:16:02 DEBUG 05:16:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:16:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:16:02 DEBUG "0" 05:16:02 DEBUG "value": [ 05:16:02 DEBUG 05:16:02 DEBUG }, 05:16:02 DEBUG 05:16:02 DEBUG }, 05:16:02 DEBUG }, 05:16:02 DEBUG ] 05:16:02 DEBUG 1735362962, 05:16:02 DEBUG 05:16:02 DEBUG "value": [ 05:16:02 DEBUG 05:16:02 DEBUG "value": [ 05:16:02 DEBUG "value": [ 05:16:02 DEBUG } 05:16:02 DEBUG "1221515.4000000001" 05:16:02 DEBUG [...] 05:16:02 DEBUG 1735362961, 05:16:02 DEBUG [...] 05:16:02 DEBUG 1735362961, 05:16:02 DEBUG 1735362962, 05:16:02 DEBUG ] 05:16:02 DEBUG ] 05:16:02 DEBUG 05:16:02 DEBUG "0.001242171637784911" 05:16:02 DEBUG 05:16:02 DEBUG "689.5446176607386" 05:16:02 DEBUG "0.036675001890387804" 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG 05:16:02 DEBUG ] 05:16:02 DEBUG 05:16:02 DEBUG ] 05:16:02 DEBUG ] 05:16:02 DEBUG } 05:16:02 DEBUG ] 05:16:02 DEBUG 05:16:02 DEBUG } 05:16:02 DEBUG 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG "instance": "10.142.0.51:9100" 05:16:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:16:02 DEBUG ] 05:16:02 DEBUG ] 05:16:02 DEBUG ] 05:16:02 DEBUG }, 05:16:02 DEBUG }, 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG "value": [ 05:16:02 DEBUG "value": [ 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG 1735362962, 05:16:02 DEBUG 1735362962, 05:16:02 DEBUG "21162.66666666667" 05:16:02 DEBUG "14880858112" 05:16:02 DEBUG ] 05:16:02 DEBUG ] 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG ] 05:16:02 DEBUG ] 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:02 DEBUG } 05:16:31 INFO 05:16:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:16:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:16:31 INFO [loop_until]: OK (rc = 0) 05:16:31 DEBUG --- stdout --- 05:16:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 261m 13377Mi rcs-687d95c5b-qs2xv 20m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:16:31 DEBUG --- stderr --- 05:16:31 DEBUG 05:16:33 INFO 05:16:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:16:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:16:33 INFO [loop_until]: OK (rc = 0) 05:16:33 DEBUG --- stdout --- 05:16:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 172m 1% 3436Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 683m 2% 18869Mi 66% 05:16:33 DEBUG --- stderr --- 05:16:33 DEBUG 05:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363020" 05:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363020" 05:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363020" 05:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363020" 05:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363020" 05:17:00 INFO [http_cmd]: http status code OK 05:17:00 DEBUG --- status code --- 05:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:00 DEBUG --- http response --- 05:17:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:00 INFO [http_cmd]: http status code OK 05:17:00 DEBUG --- status code --- 05:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:00 DEBUG --- http response --- 05:17:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:00 INFO [http_cmd]: http status code OK 05:17:00 DEBUG --- status code --- 05:17:00 INFO [http_cmd]: http status code OK 05:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:00 INFO [http_cmd]: http status code OK 05:17:00 DEBUG --- status code --- 05:17:00 DEBUG --- http response --- 05:17:00 DEBUG --- status code --- 05:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:00 DEBUG --- http response --- 05:17:00 DEBUG --- http response --- 05:17:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 ERROR [http_cmd]: ERROR 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 ERROR --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 ERROR --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG --- http response --- 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363021" 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:17:01 DEBUG ----- output ----- 05:17:01 DEBUG { 05:17:01 DEBUG "status": "success", 05:17:01 DEBUG "data": { 05:17:01 DEBUG "resultType": "vector", 05:17:01 DEBUG "result": [ 05:17:01 DEBUG { 05:17:01 DEBUG "metric": { 05:17:01 DEBUG "pod": "ds-idrepo-0" 05:17:01 DEBUG }, 05:17:01 DEBUG "value": [ 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG [...] 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:17:01 DEBUG }, 05:17:01 DEBUG "value": [ 05:17:01 DEBUG 1735363021, 05:17:01 DEBUG "831074304" 05:17:01 DEBUG ] 05:17:01 DEBUG } 05:17:01 DEBUG ] 05:17:01 DEBUG } 05:17:01 DEBUG } 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363021" 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:17:01 DEBUG ----- output ----- 05:17:01 DEBUG { 05:17:01 DEBUG "status": "success", 05:17:01 DEBUG "data": { 05:17:01 DEBUG "resultType": "vector", 05:17:01 DEBUG "result": [ 05:17:01 DEBUG { 05:17:01 DEBUG "metric": { 05:17:01 DEBUG "instance": "10.142.0.11:9100" 05:17:01 DEBUG }, 05:17:01 DEBUG "value": [ 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG [...] 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG "instance": "10.142.0.51:9100" 05:17:01 DEBUG }, 05:17:01 DEBUG "value": [ 05:17:01 DEBUG 1735363021, 05:17:01 DEBUG "0.0009666666666703352" 05:17:01 DEBUG ] 05:17:01 DEBUG } 05:17:01 DEBUG ] 05:17:01 DEBUG } 05:17:01 DEBUG } 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363021" 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:17:01 DEBUG ----- output ----- 05:17:01 DEBUG { 05:17:01 DEBUG "status": "success", 05:17:01 DEBUG "data": { 05:17:01 DEBUG "resultType": "vector", 05:17:01 DEBUG "result": [ 05:17:01 DEBUG { 05:17:01 DEBUG "metric": { 05:17:01 DEBUG "instance": "10.142.0.11:9100" 05:17:01 DEBUG }, 05:17:01 DEBUG "value": [ 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG [...] 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG "instance": "10.142.0.51:9100" 05:17:01 DEBUG }, 05:17:01 DEBUG "value": [ 05:17:01 DEBUG 1735363021, 05:17:01 DEBUG "0.0009999999999995644" 05:17:01 DEBUG ] 05:17:01 DEBUG } 05:17:01 DEBUG ] 05:17:01 DEBUG } 05:17:01 DEBUG } 05:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363021" 05:17:01 INFO [http_cmd]: http status code OK 05:17:01 DEBUG --- status code --- 05:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:01 DEBUG --- http response --- 05:17:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:17:01 DEBUG ----- output ----- 05:17:01 DEBUG { 05:17:01 DEBUG "status": "success", 05:17:01 DEBUG "data": { 05:17:01 DEBUG "resultType": "vector", 05:17:01 DEBUG "result": [ 05:17:01 DEBUG { 05:17:01 DEBUG "metric": { 05:17:01 DEBUG "instance": "10.142.0.11:9100" 05:17:01 DEBUG }, 05:17:01 DEBUG "value": [ 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG [...] 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG 05:17:01 DEBUG "instance": "10.142.0.51:9100" 05:17:01 DEBUG }, 05:17:01 DEBUG "value": [ 05:17:01 DEBUG 1735363021, 05:17:01 DEBUG "1211909.966666667" 05:17:01 DEBUG ] 05:17:01 DEBUG } 05:17:01 DEBUG ] 05:17:01 DEBUG } 05:17:01 DEBUG } 05:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363022" 05:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363022" 05:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363022" 05:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363022" 05:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363022" 05:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363022" 05:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363022" 05:17:02 INFO [http_cmd]: http status code OK 05:17:02 DEBUG --- status code --- 05:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:02 DEBUG --- http response --- 05:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:17:02 DEBUG ----- output ----- 05:17:02 DEBUG { 05:17:02 DEBUG "status": "success", 05:17:02 DEBUG "data": { 05:17:02 DEBUG "resultType": "vector", 05:17:02 DEBUG "result": [ 05:17:02 DEBUG { 05:17:02 DEBUG "metric": { 05:17:02 DEBUG "pod": "ds-idrepo-0" 05:17:02 DEBUG }, 05:17:02 DEBUG "value": [ 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG [...] 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:17:02 DEBUG }, 05:17:02 DEBUG "value": [ 05:17:02 DEBUG 1735363022, 05:17:02 DEBUG "0" 05:17:02 DEBUG ] 05:17:02 DEBUG } 05:17:02 DEBUG ] 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:02 INFO [http_cmd]: http status code OK 05:17:02 INFO [http_cmd]: http status code OK 05:17:02 DEBUG --- status code --- 05:17:02 DEBUG --- status code --- 05:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:02 DEBUG --- http response --- 05:17:02 DEBUG --- http response --- 05:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:17:02 DEBUG ----- output ----- 05:17:02 DEBUG ----- output ----- 05:17:02 DEBUG { 05:17:02 DEBUG { 05:17:02 DEBUG "status": "success", 05:17:02 DEBUG "status": "success", 05:17:02 DEBUG "data": { 05:17:02 DEBUG "data": { 05:17:02 DEBUG "resultType": "vector", 05:17:02 DEBUG "resultType": "vector", 05:17:02 DEBUG "result": [ 05:17:02 DEBUG "result": [ 05:17:02 DEBUG { 05:17:02 DEBUG { 05:17:02 DEBUG "metric": { 05:17:02 DEBUG "metric": { 05:17:02 DEBUG "pod": "ds-idrepo-0" 05:17:02 DEBUG "pod": "ds-idrepo-0" 05:17:02 DEBUG }, 05:17:02 DEBUG }, 05:17:02 DEBUG "value": [ 05:17:02 DEBUG "value": [ 05:17:02 INFO [http_cmd]: http status code OK 05:17:02 INFO [http_cmd]: http status code OK 05:17:02 DEBUG 05:17:02 INFO [http_cmd]: http status code OK 05:17:02 INFO [http_cmd]: http status code OK 05:17:02 DEBUG 05:17:02 DEBUG --- status code --- 05:17:02 DEBUG --- status code --- 05:17:02 DEBUG 05:17:02 DEBUG --- status code --- 05:17:02 DEBUG --- status code --- 05:17:02 DEBUG 05:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:02 DEBUG 05:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:02 DEBUG 05:17:02 DEBUG --- http response --- 05:17:02 DEBUG --- http response --- 05:17:02 DEBUG [...] 05:17:02 DEBUG --- http response --- 05:17:02 DEBUG --- http response --- 05:17:02 DEBUG [...] 05:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:17:02 DEBUG 05:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:17:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:17:02 DEBUG 05:17:02 DEBUG ----- output ----- 05:17:02 DEBUG ----- output ----- 05:17:02 DEBUG 05:17:02 DEBUG ----- output ----- 05:17:02 DEBUG ----- output ----- 05:17:02 DEBUG 05:17:02 DEBUG { 05:17:02 DEBUG { 05:17:02 DEBUG 05:17:02 DEBUG { 05:17:02 DEBUG { 05:17:02 DEBUG 05:17:02 DEBUG "status": "success", 05:17:02 DEBUG "status": "success", 05:17:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:17:02 DEBUG "status": "success", 05:17:02 DEBUG "status": "success", 05:17:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:17:02 DEBUG "data": { 05:17:02 DEBUG "data": { 05:17:02 DEBUG }, 05:17:02 DEBUG "data": { 05:17:02 DEBUG "data": { 05:17:02 DEBUG }, 05:17:02 DEBUG "resultType": "vector", 05:17:02 DEBUG "resultType": "vector", 05:17:02 DEBUG "value": [ 05:17:02 DEBUG "resultType": "vector", 05:17:02 DEBUG "resultType": "vector", 05:17:02 DEBUG "value": [ 05:17:02 DEBUG "result": [ 05:17:02 DEBUG "result": [ 05:17:02 DEBUG 1735363022, 05:17:02 DEBUG "result": [ 05:17:02 DEBUG "result": [ 05:17:02 DEBUG 1735363022, 05:17:02 DEBUG { 05:17:02 DEBUG { 05:17:02 DEBUG "0.003683360408781207" 05:17:02 DEBUG { 05:17:02 DEBUG { 05:17:02 DEBUG "1180.260076953456" 05:17:02 DEBUG "metric": { 05:17:02 DEBUG "metric": { 05:17:02 DEBUG ] 05:17:02 DEBUG "metric": { 05:17:02 DEBUG "metric": { 05:17:02 DEBUG ] 05:17:02 DEBUG "instance": "10.142.0.11:9100" 05:17:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:17:02 DEBUG } 05:17:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:17:02 DEBUG "instance": "10.142.0.11:9100" 05:17:02 DEBUG } 05:17:02 DEBUG }, 05:17:02 DEBUG }, 05:17:02 DEBUG ] 05:17:02 DEBUG }, 05:17:02 DEBUG }, 05:17:02 DEBUG ] 05:17:02 DEBUG "value": [ 05:17:02 DEBUG "value": [ 05:17:02 DEBUG } 05:17:02 DEBUG "value": [ 05:17:02 DEBUG "value": [ 05:17:02 DEBUG } 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG } 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG } 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG [...] 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG [...] 05:17:02 DEBUG [...] 05:17:02 DEBUG [...] 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG "instance": "10.142.0.51:9100" 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG 05:17:02 DEBUG }, 05:17:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:17:02 DEBUG "instance": "10.142.0.51:9100" 05:17:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:17:02 DEBUG "value": [ 05:17:02 DEBUG }, 05:17:02 DEBUG }, 05:17:02 DEBUG }, 05:17:02 DEBUG 1735363022, 05:17:02 DEBUG "value": [ 05:17:02 DEBUG "value": [ 05:17:02 DEBUG "value": [ 05:17:02 DEBUG "18022.4" 05:17:02 DEBUG 1735363022, 05:17:02 DEBUG 1735363022, 05:17:02 DEBUG 1735363022, 05:17:02 DEBUG ] 05:17:02 DEBUG "14878945280" 05:17:02 DEBUG "1230901.2000000004" 05:17:02 DEBUG "0.6555960530149183" 05:17:02 DEBUG } 05:17:02 DEBUG ] 05:17:02 DEBUG ] 05:17:02 DEBUG ] 05:17:02 DEBUG ] 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:02 DEBUG ] 05:17:02 DEBUG ] 05:17:02 DEBUG ] 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:02 DEBUG } 05:17:31 INFO 05:17:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:17:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:17:31 INFO [loop_until]: OK (rc = 0) 05:17:31 DEBUG --- stdout --- 05:17:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 76m 13377Mi rcs-687d95c5b-qs2xv 76m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:17:31 DEBUG --- stderr --- 05:17:31 DEBUG 05:17:33 INFO 05:17:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:17:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:17:33 INFO [loop_until]: OK (rc = 0) 05:17:33 DEBUG --- stdout --- 05:17:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 164m 1% 3441Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 129m 0% 18872Mi 66% 05:17:33 DEBUG --- stderr --- 05:17:33 DEBUG 05:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363080" 05:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363080" 05:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363080" 05:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363080" 05:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363080" 05:18:00 INFO [http_cmd]: http status code OK 05:18:00 DEBUG --- status code --- 05:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:00 INFO [http_cmd]: http status code OK 05:18:00 INFO [http_cmd]: http status code OK 05:18:00 DEBUG --- http response --- 05:18:00 INFO [http_cmd]: http status code OK 05:18:00 INFO [http_cmd]: http status code OK 05:18:00 DEBUG --- status code --- 05:18:00 DEBUG --- status code --- 05:18:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:00 DEBUG --- status code --- 05:18:00 DEBUG --- status code --- 05:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:00 DEBUG --- http response --- 05:18:00 DEBUG --- http response --- 05:18:00 DEBUG --- http response --- 05:18:00 DEBUG --- http response --- 05:18:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 ERROR [http_cmd]: ERROR 05:18:01 ERROR --- status code --- 05:18:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:18:01 ERROR --- http response --- 05:18:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:18:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363081" 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:18:01 DEBUG ----- output ----- 05:18:01 DEBUG { 05:18:01 DEBUG "status": "success", 05:18:01 DEBUG "data": { 05:18:01 DEBUG "resultType": "vector", 05:18:01 DEBUG "result": [ 05:18:01 DEBUG { 05:18:01 DEBUG "metric": { 05:18:01 DEBUG "pod": "ds-idrepo-0" 05:18:01 DEBUG }, 05:18:01 DEBUG "value": [ 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG [...] 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:18:01 DEBUG }, 05:18:01 DEBUG "value": [ 05:18:01 DEBUG 1735363081, 05:18:01 DEBUG "831074304" 05:18:01 DEBUG ] 05:18:01 DEBUG } 05:18:01 DEBUG ] 05:18:01 DEBUG } 05:18:01 DEBUG } 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363081" 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:18:01 DEBUG ----- output ----- 05:18:01 DEBUG { 05:18:01 DEBUG "status": "success", 05:18:01 DEBUG "data": { 05:18:01 DEBUG "resultType": "vector", 05:18:01 DEBUG "result": [ 05:18:01 DEBUG { 05:18:01 DEBUG "metric": { 05:18:01 DEBUG "instance": "10.142.0.11:9100" 05:18:01 DEBUG }, 05:18:01 DEBUG "value": [ 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG [...] 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG "instance": "10.142.0.51:9100" 05:18:01 DEBUG }, 05:18:01 DEBUG "value": [ 05:18:01 DEBUG 1735363081, 05:18:01 DEBUG "0.0005999999999971807" 05:18:01 DEBUG ] 05:18:01 DEBUG } 05:18:01 DEBUG ] 05:18:01 DEBUG } 05:18:01 DEBUG } 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363081" 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:18:01 DEBUG ----- output ----- 05:18:01 DEBUG { 05:18:01 DEBUG "status": "success", 05:18:01 DEBUG "data": { 05:18:01 DEBUG "resultType": "vector", 05:18:01 DEBUG "result": [ 05:18:01 DEBUG { 05:18:01 DEBUG "metric": { 05:18:01 DEBUG "instance": "10.142.0.11:9100" 05:18:01 DEBUG }, 05:18:01 DEBUG "value": [ 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG [...] 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG "instance": "10.142.0.51:9100" 05:18:01 DEBUG }, 05:18:01 DEBUG "value": [ 05:18:01 DEBUG 1735363081, 05:18:01 DEBUG "0.00033333333333303023" 05:18:01 DEBUG ] 05:18:01 DEBUG } 05:18:01 DEBUG ] 05:18:01 DEBUG } 05:18:01 DEBUG } 05:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363081" 05:18:01 INFO [http_cmd]: http status code OK 05:18:01 DEBUG --- status code --- 05:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:01 DEBUG --- http response --- 05:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:18:01 DEBUG ----- output ----- 05:18:01 DEBUG { 05:18:01 DEBUG "status": "success", 05:18:01 DEBUG "data": { 05:18:01 DEBUG "resultType": "vector", 05:18:01 DEBUG "result": [ 05:18:01 DEBUG { 05:18:01 DEBUG "metric": { 05:18:01 DEBUG "instance": "10.142.0.11:9100" 05:18:01 DEBUG }, 05:18:01 DEBUG "value": [ 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG [...] 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG 05:18:01 DEBUG "instance": "10.142.0.51:9100" 05:18:01 DEBUG }, 05:18:01 DEBUG "value": [ 05:18:01 DEBUG 1735363081, 05:18:01 DEBUG "1221660.5000000002" 05:18:01 DEBUG ] 05:18:01 DEBUG } 05:18:01 DEBUG ] 05:18:01 DEBUG } 05:18:01 DEBUG } 05:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363082" 05:18:02 INFO [http_cmd]: http status code OK 05:18:02 DEBUG --- status code --- 05:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:02 DEBUG --- http response --- 05:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363082" 05:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:18:02 DEBUG ----- output ----- 05:18:02 DEBUG { 05:18:02 DEBUG "status": "success", 05:18:02 DEBUG "data": { 05:18:02 DEBUG "resultType": "vector", 05:18:02 DEBUG "result": [ 05:18:02 DEBUG { 05:18:02 DEBUG "metric": { 05:18:02 DEBUG "pod": "ds-idrepo-0" 05:18:02 DEBUG }, 05:18:02 DEBUG "value": [ 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG [...] 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363082" 05:18:02 DEBUG 05:18:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:18:02 DEBUG }, 05:18:02 DEBUG "value": [ 05:18:02 DEBUG 1735363082, 05:18:02 DEBUG "0" 05:18:02 DEBUG ] 05:18:02 DEBUG } 05:18:02 DEBUG ] 05:18:02 DEBUG } 05:18:02 DEBUG } 05:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363082" 05:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363082" 05:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363082" 05:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363082" 05:18:02 INFO [http_cmd]: http status code OK 05:18:02 DEBUG --- status code --- 05:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:02 DEBUG --- http response --- 05:18:02 INFO [http_cmd]: http status code OK 05:18:02 DEBUG --- status code --- 05:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:18:02 DEBUG --- http response --- 05:18:02 DEBUG ----- output ----- 05:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:18:02 DEBUG { 05:18:02 DEBUG ----- output ----- 05:18:02 DEBUG "status": "success", 05:18:02 DEBUG { 05:18:02 DEBUG "data": { 05:18:02 DEBUG "status": "success", 05:18:02 DEBUG "resultType": "vector", 05:18:02 DEBUG "data": { 05:18:02 DEBUG "result": [ 05:18:02 DEBUG "resultType": "vector", 05:18:02 DEBUG { 05:18:02 DEBUG "result": [ 05:18:02 DEBUG "metric": { 05:18:02 DEBUG { 05:18:02 DEBUG "pod": "ds-idrepo-0" 05:18:02 DEBUG "metric": { 05:18:02 DEBUG }, 05:18:02 DEBUG "pod": "ds-idrepo-0" 05:18:02 DEBUG "value": [ 05:18:02 DEBUG }, 05:18:02 DEBUG 05:18:02 DEBUG "value": [ 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG [...] 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG [...] 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:18:02 DEBUG 05:18:02 DEBUG }, 05:18:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:18:02 DEBUG "value": [ 05:18:02 DEBUG }, 05:18:02 DEBUG 1735363082, 05:18:02 DEBUG "662.555759903427" 05:18:02 DEBUG "value": [ 05:18:02 DEBUG ] 05:18:02 INFO [http_cmd]: http status code OK 05:18:02 DEBUG 1735363082, 05:18:02 DEBUG } 05:18:02 DEBUG --- status code --- 05:18:02 DEBUG "0.0011885623518188494" 05:18:02 DEBUG ] 05:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:02 DEBUG ] 05:18:02 DEBUG } 05:18:02 DEBUG --- http response --- 05:18:02 DEBUG } 05:18:02 DEBUG } 05:18:02 INFO [http_cmd]: http status code OK 05:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:18:02 INFO [http_cmd]: http status code OK 05:18:02 INFO [http_cmd]: http status code OK 05:18:02 DEBUG ] 05:18:02 DEBUG --- status code --- 05:18:02 DEBUG ----- output ----- 05:18:02 DEBUG --- status code --- 05:18:02 DEBUG --- status code --- 05:18:02 DEBUG } 05:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:02 DEBUG { 05:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:02 DEBUG } 05:18:02 DEBUG --- http response --- 05:18:02 DEBUG "status": "success", 05:18:02 DEBUG --- http response --- 05:18:02 DEBUG --- http response --- 05:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:18:02 DEBUG "data": { 05:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:18:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:18:02 DEBUG ----- output ----- 05:18:02 DEBUG "resultType": "vector", 05:18:02 DEBUG ----- output ----- 05:18:02 DEBUG ----- output ----- 05:18:02 DEBUG { 05:18:02 DEBUG "result": [ 05:18:02 DEBUG { 05:18:02 DEBUG { 05:18:02 DEBUG "status": "success", 05:18:02 DEBUG { 05:18:02 DEBUG "status": "success", 05:18:02 DEBUG "status": "success", 05:18:02 DEBUG "data": { 05:18:02 DEBUG "metric": { 05:18:02 DEBUG "data": { 05:18:02 DEBUG "data": { 05:18:02 DEBUG "resultType": "vector", 05:18:02 DEBUG "instance": "10.142.0.11:9100" 05:18:02 DEBUG "resultType": "vector", 05:18:02 DEBUG "resultType": "vector", 05:18:02 DEBUG "result": [ 05:18:02 DEBUG { 05:18:02 DEBUG "result": [ 05:18:02 DEBUG "result": [ 05:18:02 DEBUG }, 05:18:02 DEBUG "metric": { 05:18:02 DEBUG { 05:18:02 DEBUG { 05:18:02 DEBUG "value": [ 05:18:02 DEBUG "instance": "10.142.0.11:9100" 05:18:02 DEBUG "metric": { 05:18:02 DEBUG "metric": { 05:18:02 DEBUG 05:18:02 DEBUG }, 05:18:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:18:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:18:02 DEBUG 05:18:02 DEBUG "value": [ 05:18:02 DEBUG }, 05:18:02 DEBUG }, 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG "value": [ 05:18:02 DEBUG "value": [ 05:18:02 DEBUG [...] 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG [...] 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG [...] 05:18:02 DEBUG [...] 05:18:02 DEBUG "instance": "10.142.0.51:9100" 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG }, 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG "value": [ 05:18:02 DEBUG "instance": "10.142.0.51:9100" 05:18:02 DEBUG 05:18:02 DEBUG 05:18:02 DEBUG 1735363082, 05:18:02 DEBUG }, 05:18:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:18:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:18:02 DEBUG "36590.93333333334" 05:18:02 DEBUG "value": [ 05:18:02 DEBUG }, 05:18:02 DEBUG }, 05:18:02 DEBUG ] 05:18:02 DEBUG 1735363082, 05:18:02 DEBUG "value": [ 05:18:02 DEBUG "value": [ 05:18:02 DEBUG } 05:18:02 DEBUG "1240709.1" 05:18:02 DEBUG 1735363082, 05:18:02 DEBUG 1735363082, 05:18:02 DEBUG ] 05:18:02 DEBUG ] 05:18:02 DEBUG "14881161216" 05:18:02 DEBUG "0.10067889728435704" 05:18:02 DEBUG } 05:18:02 DEBUG } 05:18:02 DEBUG ] 05:18:02 DEBUG ] 05:18:02 DEBUG } 05:18:02 DEBUG ] 05:18:02 DEBUG } 05:18:02 DEBUG } 05:18:02 DEBUG } 05:18:02 DEBUG ] 05:18:02 DEBUG ] 05:18:02 DEBUG } 05:18:02 DEBUG } 05:18:02 DEBUG } 05:18:02 DEBUG } 05:18:02 DEBUG } 05:18:31 INFO 05:18:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:18:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:18:32 INFO [loop_until]: OK (rc = 0) 05:18:32 DEBUG --- stdout --- 05:18:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 533Mi ds-idrepo-0 56m 13378Mi rcs-687d95c5b-qs2xv 84m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:18:32 DEBUG --- stderr --- 05:18:32 DEBUG 05:18:33 INFO 05:18:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:18:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:18:33 INFO [loop_until]: OK (rc = 0) 05:18:33 DEBUG --- stdout --- 05:18:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 170m 1% 3426Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 129m 0% 18872Mi 66% 05:18:33 DEBUG --- stderr --- 05:18:33 DEBUG 05:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363140" 05:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363140" 05:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363140" 05:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363140" 05:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363140" 05:19:00 INFO [http_cmd]: http status code OK 05:19:00 DEBUG --- status code --- 05:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:00 DEBUG --- http response --- 05:19:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:00 INFO [http_cmd]: http status code OK 05:19:00 DEBUG --- status code --- 05:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:00 DEBUG --- http response --- 05:19:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:00 INFO [http_cmd]: http status code OK 05:19:00 DEBUG --- status code --- 05:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:00 DEBUG --- http response --- 05:19:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:00 INFO [http_cmd]: http status code OK 05:19:00 DEBUG --- status code --- 05:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:00 DEBUG --- http response --- 05:19:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:00 INFO [http_cmd]: http status code OK 05:19:00 DEBUG --- status code --- 05:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:00 DEBUG --- http response --- 05:19:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 ERROR [http_cmd]: ERROR 05:19:01 ERROR --- status code --- 05:19:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:19:01 ERROR --- http response --- 05:19:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:19:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363141" 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:19:01 DEBUG ----- output ----- 05:19:01 DEBUG { 05:19:01 DEBUG "status": "success", 05:19:01 DEBUG "data": { 05:19:01 DEBUG "resultType": "vector", 05:19:01 DEBUG "result": [ 05:19:01 DEBUG { 05:19:01 DEBUG "metric": { 05:19:01 DEBUG "pod": "ds-idrepo-0" 05:19:01 DEBUG }, 05:19:01 DEBUG "value": [ 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG [...] 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:19:01 DEBUG }, 05:19:01 DEBUG "value": [ 05:19:01 DEBUG 1735363141, 05:19:01 DEBUG "831074304" 05:19:01 DEBUG ] 05:19:01 DEBUG } 05:19:01 DEBUG ] 05:19:01 DEBUG } 05:19:01 DEBUG } 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363141" 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:19:01 DEBUG ----- output ----- 05:19:01 DEBUG { 05:19:01 DEBUG "status": "success", 05:19:01 DEBUG "data": { 05:19:01 DEBUG "resultType": "vector", 05:19:01 DEBUG "result": [ 05:19:01 DEBUG { 05:19:01 DEBUG "metric": { 05:19:01 DEBUG "instance": "10.142.0.11:9100" 05:19:01 DEBUG }, 05:19:01 DEBUG "value": [ 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG [...] 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG "instance": "10.142.0.51:9100" 05:19:01 DEBUG }, 05:19:01 DEBUG "value": [ 05:19:01 DEBUG 1735363141, 05:19:01 DEBUG "0.0006333333333410946" 05:19:01 DEBUG ] 05:19:01 DEBUG } 05:19:01 DEBUG ] 05:19:01 DEBUG } 05:19:01 DEBUG } 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363141" 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:19:01 DEBUG ----- output ----- 05:19:01 DEBUG { 05:19:01 DEBUG "status": "success", 05:19:01 DEBUG "data": { 05:19:01 DEBUG "resultType": "vector", 05:19:01 DEBUG "result": [ 05:19:01 DEBUG { 05:19:01 DEBUG "metric": { 05:19:01 DEBUG "instance": "10.142.0.11:9100" 05:19:01 DEBUG }, 05:19:01 DEBUG "value": [ 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG [...] 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG "instance": "10.142.0.51:9100" 05:19:01 DEBUG }, 05:19:01 DEBUG "value": [ 05:19:01 DEBUG 1735363141, 05:19:01 DEBUG "0" 05:19:01 DEBUG ] 05:19:01 DEBUG } 05:19:01 DEBUG ] 05:19:01 DEBUG } 05:19:01 DEBUG } 05:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363141" 05:19:01 INFO [http_cmd]: http status code OK 05:19:01 DEBUG --- status code --- 05:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:01 DEBUG --- http response --- 05:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:19:01 DEBUG ----- output ----- 05:19:01 DEBUG { 05:19:01 DEBUG "status": "success", 05:19:01 DEBUG "data": { 05:19:01 DEBUG "resultType": "vector", 05:19:01 DEBUG "result": [ 05:19:01 DEBUG { 05:19:01 DEBUG "metric": { 05:19:01 DEBUG "instance": "10.142.0.11:9100" 05:19:01 DEBUG }, 05:19:01 DEBUG "value": [ 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG [...] 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG 05:19:01 DEBUG "instance": "10.142.0.51:9100" 05:19:01 DEBUG }, 05:19:01 DEBUG "value": [ 05:19:01 DEBUG 1735363141, 05:19:01 DEBUG "1222988.066666667" 05:19:01 DEBUG ] 05:19:01 DEBUG } 05:19:01 DEBUG ] 05:19:01 DEBUG } 05:19:01 DEBUG } 05:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363142" 05:19:02 INFO [http_cmd]: http status code OK 05:19:02 DEBUG --- status code --- 05:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:02 DEBUG --- http response --- 05:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:19:02 DEBUG ----- output ----- 05:19:02 DEBUG { 05:19:02 DEBUG "status": "success", 05:19:02 DEBUG "data": { 05:19:02 DEBUG "resultType": "vector", 05:19:02 DEBUG "result": [ 05:19:02 DEBUG { 05:19:02 DEBUG "metric": { 05:19:02 DEBUG "pod": "ds-idrepo-0" 05:19:02 DEBUG }, 05:19:02 DEBUG "value": [ 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG [...] 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:19:02 DEBUG }, 05:19:02 DEBUG "value": [ 05:19:02 DEBUG 1735363142, 05:19:02 DEBUG "0" 05:19:02 DEBUG ] 05:19:02 DEBUG } 05:19:02 DEBUG ] 05:19:02 DEBUG } 05:19:02 DEBUG } 05:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363142" 05:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363142" 05:19:02 INFO [http_cmd]: http status code OK 05:19:02 DEBUG --- status code --- 05:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363142" 05:19:02 DEBUG --- http response --- 05:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:19:02 DEBUG ----- output ----- 05:19:02 DEBUG { 05:19:02 DEBUG "status": "success", 05:19:02 DEBUG "data": { 05:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363142" 05:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363142" 05:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363142" 05:19:02 DEBUG "resultType": "vector", 05:19:02 INFO [http_cmd]: http status code OK 05:19:02 DEBUG "result": [ 05:19:02 DEBUG --- status code --- 05:19:02 DEBUG { 05:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:02 DEBUG "metric": { 05:19:02 DEBUG --- http response --- 05:19:02 DEBUG "pod": "ds-idrepo-0" 05:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:19:02 DEBUG }, 05:19:02 DEBUG ----- output ----- 05:19:02 DEBUG "value": [ 05:19:02 DEBUG { 05:19:02 DEBUG 05:19:02 DEBUG "status": "success", 05:19:02 DEBUG 05:19:02 DEBUG "data": { 05:19:02 DEBUG 05:19:02 DEBUG "resultType": "vector", 05:19:02 DEBUG [...] 05:19:02 DEBUG "result": [ 05:19:02 DEBUG 05:19:02 DEBUG { 05:19:02 DEBUG 05:19:02 DEBUG "metric": { 05:19:02 DEBUG 05:19:02 DEBUG "pod": "ds-idrepo-0" 05:19:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:19:02 DEBUG }, 05:19:02 DEBUG }, 05:19:02 DEBUG "value": [ 05:19:02 DEBUG "value": [ 05:19:02 DEBUG 05:19:02 DEBUG 1735363142, 05:19:02 DEBUG 05:19:02 DEBUG "861.2009407598912" 05:19:02 DEBUG 05:19:02 DEBUG ] 05:19:02 DEBUG [...] 05:19:02 DEBUG } 05:19:02 DEBUG 05:19:02 DEBUG ] 05:19:02 DEBUG 05:19:02 DEBUG } 05:19:02 DEBUG 05:19:02 DEBUG } 05:19:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:19:02 DEBUG }, 05:19:02 DEBUG "value": [ 05:19:02 DEBUG 1735363142, 05:19:02 DEBUG "0.001291411188833044" 05:19:02 DEBUG ] 05:19:02 DEBUG } 05:19:02 DEBUG ] 05:19:02 DEBUG } 05:19:02 DEBUG } 05:19:02 INFO [http_cmd]: http status code OK 05:19:02 INFO [http_cmd]: http status code OK 05:19:02 DEBUG --- status code --- 05:19:02 DEBUG --- status code --- 05:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:02 DEBUG --- http response --- 05:19:02 DEBUG --- http response --- 05:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:19:02 DEBUG ----- output ----- 05:19:02 DEBUG ----- output ----- 05:19:02 DEBUG { 05:19:02 DEBUG { 05:19:02 DEBUG "status": "success", 05:19:02 DEBUG "status": "success", 05:19:02 DEBUG "data": { 05:19:02 DEBUG "data": { 05:19:02 DEBUG "resultType": "vector", 05:19:02 DEBUG "resultType": "vector", 05:19:02 DEBUG "result": [ 05:19:02 DEBUG "result": [ 05:19:02 DEBUG { 05:19:02 DEBUG { 05:19:02 DEBUG "metric": { 05:19:02 DEBUG "metric": { 05:19:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:19:02 INFO [http_cmd]: http status code OK 05:19:02 DEBUG "instance": "10.142.0.11:9100" 05:19:02 DEBUG }, 05:19:02 INFO [http_cmd]: http status code OK 05:19:02 DEBUG --- status code --- 05:19:02 DEBUG }, 05:19:02 DEBUG "value": [ 05:19:02 DEBUG --- status code --- 05:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:02 DEBUG "value": [ 05:19:02 DEBUG 05:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:02 DEBUG --- http response --- 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG --- http response --- 05:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG ----- output ----- 05:19:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:19:02 DEBUG 05:19:02 DEBUG [...] 05:19:02 DEBUG { 05:19:02 DEBUG ----- output ----- 05:19:02 DEBUG [...] 05:19:02 DEBUG 05:19:02 DEBUG "status": "success", 05:19:02 DEBUG { 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG "data": { 05:19:02 DEBUG "status": "success", 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG "resultType": "vector", 05:19:02 DEBUG "data": { 05:19:02 DEBUG 05:19:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:19:02 DEBUG "result": [ 05:19:02 DEBUG "resultType": "vector", 05:19:02 DEBUG "instance": "10.142.0.51:9100" 05:19:02 DEBUG }, 05:19:02 DEBUG { 05:19:02 DEBUG "result": [ 05:19:02 DEBUG }, 05:19:02 DEBUG "value": [ 05:19:02 DEBUG "metric": { 05:19:02 DEBUG { 05:19:02 DEBUG "value": [ 05:19:02 DEBUG 1735363142, 05:19:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:19:02 DEBUG "metric": { 05:19:02 DEBUG 1735363142, 05:19:02 DEBUG "14880370688" 05:19:02 DEBUG }, 05:19:02 DEBUG "instance": "10.142.0.11:9100" 05:19:02 DEBUG "22937.600000000006" 05:19:02 DEBUG ] 05:19:02 DEBUG "value": [ 05:19:02 DEBUG }, 05:19:02 DEBUG ] 05:19:02 DEBUG } 05:19:02 DEBUG 05:19:02 DEBUG "value": [ 05:19:02 DEBUG } 05:19:02 DEBUG ] 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG ] 05:19:02 DEBUG } 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG } 05:19:02 DEBUG } 05:19:02 DEBUG [...] 05:19:02 DEBUG 05:19:02 DEBUG } 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG [...] 05:19:02 DEBUG 05:19:02 DEBUG 05:19:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:19:02 DEBUG 05:19:02 DEBUG }, 05:19:02 DEBUG 05:19:02 DEBUG "value": [ 05:19:02 DEBUG "instance": "10.142.0.51:9100" 05:19:02 DEBUG 1735363142, 05:19:02 DEBUG }, 05:19:02 DEBUG "0.08293064060184178" 05:19:02 DEBUG "value": [ 05:19:02 DEBUG ] 05:19:02 DEBUG 1735363142, 05:19:02 DEBUG } 05:19:02 DEBUG "1242036.066666667" 05:19:02 DEBUG ] 05:19:02 DEBUG ] 05:19:02 DEBUG } 05:19:02 DEBUG } 05:19:02 DEBUG } 05:19:02 DEBUG ] 05:19:02 DEBUG } 05:19:02 DEBUG } 05:19:32 INFO 05:19:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:19:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:19:32 INFO [loop_until]: OK (rc = 0) 05:19:32 DEBUG --- stdout --- 05:19:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 77m 13378Mi rcs-687d95c5b-qs2xv 66m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:19:32 DEBUG --- stderr --- 05:19:32 DEBUG 05:19:33 INFO 05:19:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:19:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:19:34 INFO [loop_until]: OK (rc = 0) 05:19:34 DEBUG --- stdout --- 05:19:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 200m 1% 3439Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 174m 0% 18870Mi 66% 05:19:34 DEBUG --- stderr --- 05:19:34 DEBUG 05:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363200" 05:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363200" 05:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363200" 05:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363200" 05:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363200" 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 ERROR [http_cmd]: ERROR 05:20:01 ERROR --- status code --- 05:20:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:20:01 ERROR --- http response --- 05:20:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:20:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363201" 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:20:01 DEBUG ----- output ----- 05:20:01 DEBUG { 05:20:01 DEBUG "status": "success", 05:20:01 DEBUG "data": { 05:20:01 DEBUG "resultType": "vector", 05:20:01 DEBUG "result": [ 05:20:01 DEBUG { 05:20:01 DEBUG "metric": { 05:20:01 DEBUG "pod": "ds-idrepo-0" 05:20:01 DEBUG }, 05:20:01 DEBUG "value": [ 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG [...] 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:20:01 DEBUG }, 05:20:01 DEBUG "value": [ 05:20:01 DEBUG 1735363201, 05:20:01 DEBUG "831074304" 05:20:01 DEBUG ] 05:20:01 DEBUG } 05:20:01 DEBUG ] 05:20:01 DEBUG } 05:20:01 DEBUG } 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363201" 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:20:01 DEBUG ----- output ----- 05:20:01 DEBUG { 05:20:01 DEBUG "status": "success", 05:20:01 DEBUG "data": { 05:20:01 DEBUG "resultType": "vector", 05:20:01 DEBUG "result": [ 05:20:01 DEBUG { 05:20:01 DEBUG "metric": { 05:20:01 DEBUG "instance": "10.142.0.11:9100" 05:20:01 DEBUG }, 05:20:01 DEBUG "value": [ 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG [...] 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG "instance": "10.142.0.51:9100" 05:20:01 DEBUG }, 05:20:01 DEBUG "value": [ 05:20:01 DEBUG 1735363201, 05:20:01 DEBUG "0.00040000000000191" 05:20:01 DEBUG ] 05:20:01 DEBUG } 05:20:01 DEBUG ] 05:20:01 DEBUG } 05:20:01 DEBUG } 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363201" 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:20:01 DEBUG ----- output ----- 05:20:01 DEBUG { 05:20:01 DEBUG "status": "success", 05:20:01 DEBUG "data": { 05:20:01 DEBUG "resultType": "vector", 05:20:01 DEBUG "result": [ 05:20:01 DEBUG { 05:20:01 DEBUG "metric": { 05:20:01 DEBUG "instance": "10.142.0.11:9100" 05:20:01 DEBUG }, 05:20:01 DEBUG "value": [ 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG [...] 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG "instance": "10.142.0.51:9100" 05:20:01 DEBUG }, 05:20:01 DEBUG "value": [ 05:20:01 DEBUG 1735363201, 05:20:01 DEBUG "0" 05:20:01 DEBUG ] 05:20:01 DEBUG } 05:20:01 DEBUG ] 05:20:01 DEBUG } 05:20:01 DEBUG } 05:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363201" 05:20:01 INFO [http_cmd]: http status code OK 05:20:01 DEBUG --- status code --- 05:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:01 DEBUG --- http response --- 05:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:20:01 DEBUG ----- output ----- 05:20:01 DEBUG { 05:20:01 DEBUG "status": "success", 05:20:01 DEBUG "data": { 05:20:01 DEBUG "resultType": "vector", 05:20:01 DEBUG "result": [ 05:20:01 DEBUG { 05:20:01 DEBUG "metric": { 05:20:01 DEBUG "instance": "10.142.0.11:9100" 05:20:01 DEBUG }, 05:20:01 DEBUG "value": [ 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG [...] 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG 05:20:01 DEBUG "instance": "10.142.0.51:9100" 05:20:01 DEBUG }, 05:20:01 DEBUG "value": [ 05:20:01 DEBUG 1735363201, 05:20:01 DEBUG "2372683.766666667" 05:20:01 DEBUG ] 05:20:01 DEBUG } 05:20:01 DEBUG ] 05:20:01 DEBUG } 05:20:01 DEBUG } 05:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363202" 05:20:02 INFO [http_cmd]: http status code OK 05:20:02 DEBUG --- status code --- 05:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:02 DEBUG --- http response --- 05:20:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:20:02 DEBUG ----- output ----- 05:20:02 DEBUG { 05:20:02 DEBUG "status": "success", 05:20:02 DEBUG "data": { 05:20:02 DEBUG "resultType": "vector", 05:20:02 DEBUG "result": [ 05:20:02 DEBUG { 05:20:02 DEBUG "metric": { 05:20:02 DEBUG "pod": "ds-idrepo-0" 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG [...] 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 1735363202, 05:20:02 DEBUG "0" 05:20:02 DEBUG ] 05:20:02 DEBUG } 05:20:02 DEBUG ] 05:20:02 DEBUG } 05:20:02 DEBUG } 05:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363202" 05:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363202" 05:20:02 INFO [http_cmd]: http status code OK 05:20:02 DEBUG --- status code --- 05:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:02 DEBUG --- http response --- 05:20:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:20:02 DEBUG ----- output ----- 05:20:02 DEBUG { 05:20:02 DEBUG "status": "success", 05:20:02 DEBUG "data": { 05:20:02 DEBUG "resultType": "vector", 05:20:02 DEBUG "result": [ 05:20:02 DEBUG { 05:20:02 DEBUG "metric": { 05:20:02 DEBUG "pod": "ds-idrepo-0" 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG [...] 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 1735363202, 05:20:02 DEBUG "1141.2910501767954" 05:20:02 DEBUG ] 05:20:02 DEBUG } 05:20:02 DEBUG ] 05:20:02 DEBUG } 05:20:02 DEBUG } 05:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363202" 05:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363202" 05:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363202" 05:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363202" 05:20:02 INFO [http_cmd]: http status code OK 05:20:02 DEBUG --- status code --- 05:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:02 DEBUG --- http response --- 05:20:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:20:02 DEBUG ----- output ----- 05:20:02 DEBUG { 05:20:02 DEBUG "status": "success", 05:20:02 DEBUG "data": { 05:20:02 DEBUG "resultType": "vector", 05:20:02 DEBUG "result": [ 05:20:02 DEBUG { 05:20:02 DEBUG "metric": { 05:20:02 DEBUG "pod": "ds-idrepo-0" 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG [...] 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 1735363202, 05:20:02 DEBUG "0.0012964332772741304" 05:20:02 DEBUG ] 05:20:02 DEBUG } 05:20:02 DEBUG ] 05:20:02 DEBUG } 05:20:02 DEBUG } 05:20:02 INFO [http_cmd]: http status code OK 05:20:02 DEBUG --- status code --- 05:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:02 DEBUG --- http response --- 05:20:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:20:02 DEBUG ----- output ----- 05:20:02 DEBUG { 05:20:02 DEBUG "status": "success", 05:20:02 DEBUG "data": { 05:20:02 DEBUG "resultType": "vector", 05:20:02 DEBUG "result": [ 05:20:02 DEBUG { 05:20:02 DEBUG "metric": { 05:20:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG [...] 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 1735363202, 05:20:02 DEBUG "14879469568" 05:20:02 DEBUG ] 05:20:02 INFO [http_cmd]: http status code OK 05:20:02 DEBUG } 05:20:02 DEBUG --- status code --- 05:20:02 DEBUG ] 05:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:02 DEBUG } 05:20:02 DEBUG --- http response --- 05:20:02 DEBUG } 05:20:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:20:02 DEBUG ----- output ----- 05:20:02 DEBUG { 05:20:02 DEBUG "status": "success", 05:20:02 DEBUG "data": { 05:20:02 DEBUG "resultType": "vector", 05:20:02 DEBUG "result": [ 05:20:02 DEBUG { 05:20:02 DEBUG "metric": { 05:20:02 DEBUG "instance": "10.142.0.11:9100" 05:20:02 INFO [http_cmd]: http status code OK 05:20:02 DEBUG }, 05:20:02 DEBUG --- status code --- 05:20:02 DEBUG "value": [ 05:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:02 DEBUG 05:20:02 DEBUG --- http response --- 05:20:02 DEBUG 05:20:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:20:02 DEBUG 05:20:02 DEBUG ----- output ----- 05:20:02 INFO [http_cmd]: http status code OK 05:20:02 DEBUG [...] 05:20:02 DEBUG { 05:20:02 DEBUG --- status code --- 05:20:02 DEBUG 05:20:02 DEBUG "status": "success", 05:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:02 DEBUG 05:20:02 DEBUG "data": { 05:20:02 DEBUG --- http response --- 05:20:02 DEBUG 05:20:02 DEBUG "resultType": "vector", 05:20:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:20:02 DEBUG "instance": "10.142.0.51:9100" 05:20:02 DEBUG "result": [ 05:20:02 DEBUG ----- output ----- 05:20:02 DEBUG }, 05:20:02 DEBUG { 05:20:02 DEBUG { 05:20:02 DEBUG "value": [ 05:20:02 DEBUG "metric": { 05:20:02 DEBUG "status": "success", 05:20:02 DEBUG 1735363202, 05:20:02 DEBUG "instance": "10.142.0.11:9100" 05:20:02 DEBUG "data": { 05:20:02 DEBUG "20616.533333333336" 05:20:02 DEBUG }, 05:20:02 DEBUG "resultType": "vector", 05:20:02 DEBUG ] 05:20:02 DEBUG "value": [ 05:20:02 DEBUG "result": [ 05:20:02 DEBUG } 05:20:02 DEBUG 05:20:02 DEBUG { 05:20:02 DEBUG ] 05:20:02 DEBUG 05:20:02 DEBUG "metric": { 05:20:02 DEBUG } 05:20:02 DEBUG 05:20:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:20:02 DEBUG } 05:20:02 DEBUG [...] 05:20:02 DEBUG 05:20:02 DEBUG }, 05:20:02 DEBUG 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 05:20:02 DEBUG 05:20:02 DEBUG "instance": "10.142.0.51:9100" 05:20:02 DEBUG 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 1735363202, 05:20:02 DEBUG 05:20:02 DEBUG "2389861.366666667" 05:20:02 DEBUG [...] 05:20:02 DEBUG ] 05:20:02 DEBUG 05:20:02 DEBUG } 05:20:02 DEBUG 05:20:02 DEBUG ] 05:20:02 DEBUG 05:20:02 DEBUG } 05:20:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:20:02 DEBUG } 05:20:02 DEBUG }, 05:20:02 DEBUG "value": [ 05:20:02 DEBUG 1735363202, 05:20:02 DEBUG "0.1145144887033071" 05:20:02 DEBUG ] 05:20:02 DEBUG } 05:20:02 DEBUG ] 05:20:02 DEBUG } 05:20:02 DEBUG } 05:20:32 INFO 05:20:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:20:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:20:32 INFO [loop_until]: OK (rc = 0) 05:20:32 DEBUG --- stdout --- 05:20:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 95m 13378Mi rcs-687d95c5b-qs2xv 99m 426Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:20:32 DEBUG --- stderr --- 05:20:32 DEBUG 05:20:34 INFO 05:20:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:20:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:20:34 INFO [loop_until]: OK (rc = 0) 05:20:34 DEBUG --- stdout --- 05:20:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 195m 1% 3437Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 124m 0% 18871Mi 66% 05:20:34 DEBUG --- stderr --- 05:20:34 DEBUG 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- http response --- 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 ERROR [http_cmd]: ERROR 05:21:01 DEBUG --- status code --- 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 ERROR --- status code --- 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- status code --- 05:21:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 ERROR --- http response --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363261" 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:21:01 DEBUG ----- output ----- 05:21:01 DEBUG { 05:21:01 DEBUG "status": "success", 05:21:01 DEBUG "data": { 05:21:01 DEBUG "resultType": "vector", 05:21:01 DEBUG "result": [ 05:21:01 DEBUG { 05:21:01 DEBUG "metric": { 05:21:01 DEBUG "pod": "ds-idrepo-0" 05:21:01 DEBUG }, 05:21:01 DEBUG "value": [ 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG [...] 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:21:01 DEBUG }, 05:21:01 DEBUG "value": [ 05:21:01 DEBUG 1735363261, 05:21:01 DEBUG "831074304" 05:21:01 DEBUG ] 05:21:01 DEBUG } 05:21:01 DEBUG ] 05:21:01 DEBUG } 05:21:01 DEBUG } 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363261" 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:21:01 DEBUG ----- output ----- 05:21:01 DEBUG { 05:21:01 DEBUG "status": "success", 05:21:01 DEBUG "data": { 05:21:01 DEBUG "resultType": "vector", 05:21:01 DEBUG "result": [ 05:21:01 DEBUG { 05:21:01 DEBUG "metric": { 05:21:01 DEBUG "instance": "10.142.0.11:9100" 05:21:01 DEBUG }, 05:21:01 DEBUG "value": [ 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG [...] 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG "instance": "10.142.0.51:9100" 05:21:01 DEBUG }, 05:21:01 DEBUG "value": [ 05:21:01 DEBUG 1735363261, 05:21:01 DEBUG "0.0004666666666632107" 05:21:01 DEBUG ] 05:21:01 DEBUG } 05:21:01 DEBUG ] 05:21:01 DEBUG } 05:21:01 DEBUG } 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363261" 05:21:01 INFO [http_cmd]: http status code OK 05:21:01 DEBUG --- status code --- 05:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:01 DEBUG --- http response --- 05:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:21:01 DEBUG ----- output ----- 05:21:01 DEBUG { 05:21:01 DEBUG "status": "success", 05:21:01 DEBUG "data": { 05:21:01 DEBUG "resultType": "vector", 05:21:01 DEBUG "result": [ 05:21:01 DEBUG { 05:21:01 DEBUG "metric": { 05:21:01 DEBUG "instance": "10.142.0.11:9100" 05:21:01 DEBUG }, 05:21:01 DEBUG "value": [ 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG [...] 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG 05:21:01 DEBUG "instance": "10.142.0.51:9100" 05:21:01 DEBUG }, 05:21:01 DEBUG "value": [ 05:21:01 DEBUG 1735363261, 05:21:01 DEBUG "0" 05:21:01 DEBUG ] 05:21:01 DEBUG } 05:21:01 DEBUG ] 05:21:01 DEBUG } 05:21:01 DEBUG } 05:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363261" 05:21:02 INFO [http_cmd]: http status code OK 05:21:02 DEBUG --- status code --- 05:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:02 DEBUG --- http response --- 05:21:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:21:02 DEBUG ----- output ----- 05:21:02 DEBUG { 05:21:02 DEBUG "status": "success", 05:21:02 DEBUG "data": { 05:21:02 DEBUG "resultType": "vector", 05:21:02 DEBUG "result": [ 05:21:02 DEBUG { 05:21:02 DEBUG "metric": { 05:21:02 DEBUG "instance": "10.142.0.11:9100" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG [...] 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG "instance": "10.142.0.51:9100" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 1735363261, 05:21:02 DEBUG "2048706.0000000007" 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG } 05:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363262" 05:21:02 INFO [http_cmd]: http status code OK 05:21:02 DEBUG --- status code --- 05:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:02 DEBUG --- http response --- 05:21:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:21:02 DEBUG ----- output ----- 05:21:02 DEBUG { 05:21:02 DEBUG "status": "success", 05:21:02 DEBUG "data": { 05:21:02 DEBUG "resultType": "vector", 05:21:02 DEBUG "result": [ 05:21:02 DEBUG { 05:21:02 DEBUG "metric": { 05:21:02 DEBUG "pod": "ds-idrepo-0" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG [...] 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 1735363262, 05:21:02 DEBUG "0" 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG } 05:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363262" 05:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363262" 05:21:02 INFO [http_cmd]: http status code OK 05:21:02 DEBUG --- status code --- 05:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:02 DEBUG --- http response --- 05:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363262" 05:21:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:21:02 DEBUG ----- output ----- 05:21:02 DEBUG { 05:21:02 DEBUG "status": "success", 05:21:02 DEBUG "data": { 05:21:02 DEBUG "resultType": "vector", 05:21:02 DEBUG "result": [ 05:21:02 DEBUG { 05:21:02 DEBUG "metric": { 05:21:02 DEBUG "pod": "ds-idrepo-0" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG [...] 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 1735363262, 05:21:02 DEBUG "793.4555188279344" 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG } 05:21:02 INFO [http_cmd]: http status code OK 05:21:02 DEBUG --- status code --- 05:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:02 DEBUG --- http response --- 05:21:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:21:02 DEBUG ----- output ----- 05:21:02 DEBUG { 05:21:02 DEBUG "status": "success", 05:21:02 DEBUG "data": { 05:21:02 DEBUG "resultType": "vector", 05:21:02 DEBUG "result": [ 05:21:02 DEBUG { 05:21:02 DEBUG "metric": { 05:21:02 DEBUG "pod": "ds-idrepo-0" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG [...] 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 1735363262, 05:21:02 DEBUG "0.0012126396512984066" 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG } 05:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363262" 05:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363262" 05:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363262" 05:21:02 INFO [http_cmd]: http status code OK 05:21:02 DEBUG --- status code --- 05:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:02 DEBUG --- http response --- 05:21:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:21:02 DEBUG ----- output ----- 05:21:02 DEBUG { 05:21:02 DEBUG "status": "success", 05:21:02 DEBUG "data": { 05:21:02 DEBUG "resultType": "vector", 05:21:02 DEBUG "result": [ 05:21:02 DEBUG { 05:21:02 DEBUG "metric": { 05:21:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG [...] 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:21:02 DEBUG }, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 1735363262, 05:21:02 DEBUG "14878375936" 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG } 05:21:02 INFO [http_cmd]: http status code OK 05:21:02 DEBUG --- status code --- 05:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:02 INFO [http_cmd]: http status code OK 05:21:02 DEBUG --- http response --- 05:21:02 INFO [http_cmd]: http status code OK 05:21:02 DEBUG --- status code --- 05:21:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:21:02 DEBUG --- status code --- 05:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:02 DEBUG ----- output ----- 05:21:02 DEBUG { 05:21:02 DEBUG --- http response --- 05:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:02 DEBUG "status": "success", 05:21:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:21:02 DEBUG --- http response --- 05:21:02 DEBUG "data": { 05:21:02 DEBUG ----- output ----- 05:21:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:21:02 DEBUG "resultType": "vector", 05:21:02 DEBUG { 05:21:02 DEBUG ----- output ----- 05:21:02 DEBUG "result": [ 05:21:02 DEBUG "status": "success", 05:21:02 DEBUG { 05:21:02 DEBUG { 05:21:02 DEBUG "data": { 05:21:02 DEBUG "status": "success", 05:21:02 DEBUG "metric": { 05:21:02 DEBUG "resultType": "vector", 05:21:02 DEBUG "data": { 05:21:02 DEBUG "instance": "10.142.0.11:9100" 05:21:02 DEBUG "result": [ 05:21:02 DEBUG "resultType": "vector", 05:21:02 DEBUG }, 05:21:02 DEBUG { 05:21:02 DEBUG "result": [ 05:21:02 DEBUG "value": [ 05:21:02 DEBUG "metric": { 05:21:02 DEBUG { 05:21:02 DEBUG 05:21:02 DEBUG "instance": "10.142.0.11:9100" 05:21:02 DEBUG "metric": { 05:21:02 DEBUG 05:21:02 DEBUG }, 05:21:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:21:02 DEBUG 05:21:02 DEBUG "value": [ 05:21:02 DEBUG }, 05:21:02 DEBUG [...] 05:21:02 DEBUG 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG [...] 05:21:02 DEBUG 05:21:02 DEBUG "instance": "10.142.0.51:9100" 05:21:02 DEBUG 05:21:02 DEBUG [...] 05:21:02 DEBUG }, 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG "value": [ 05:21:02 DEBUG 05:21:02 DEBUG 05:21:02 DEBUG 1735363262, 05:21:02 DEBUG "instance": "10.142.0.51:9100" 05:21:02 DEBUG 05:21:02 DEBUG "26487.46666666667" 05:21:02 DEBUG }, 05:21:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:21:02 DEBUG ] 05:21:02 DEBUG "value": [ 05:21:02 DEBUG }, 05:21:02 DEBUG } 05:21:02 DEBUG 1735363262, 05:21:02 DEBUG "value": [ 05:21:02 DEBUG ] 05:21:02 DEBUG "2066339.8000000003" 05:21:02 DEBUG 1735363262, 05:21:02 DEBUG } 05:21:02 DEBUG ] 05:21:02 DEBUG "0.12379283956129869" 05:21:02 DEBUG } 05:21:02 DEBUG } 05:21:02 DEBUG ] 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG } 05:21:02 DEBUG ] 05:21:02 DEBUG } 05:21:02 DEBUG } 05:21:02 DEBUG } 05:21:32 INFO 05:21:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:21:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:21:32 INFO [loop_until]: OK (rc = 0) 05:21:32 DEBUG --- stdout --- 05:21:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 66m 13378Mi rcs-687d95c5b-qs2xv 88m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:21:32 DEBUG --- stderr --- 05:21:32 DEBUG 05:21:34 INFO 05:21:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:21:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:21:34 INFO [loop_until]: OK (rc = 0) 05:21:34 DEBUG --- stdout --- 05:21:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 166m 1% 3439Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 131m 0% 18873Mi 66% 05:21:34 DEBUG --- stderr --- 05:21:34 DEBUG 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG --- status code --- 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 ERROR [http_cmd]: ERROR 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- status code --- 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 ERROR --- status code --- 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- status code --- 05:22:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG --- http response --- 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 ERROR --- http response --- 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363321" 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:22:01 DEBUG ----- output ----- 05:22:01 DEBUG { 05:22:01 DEBUG "status": "success", 05:22:01 DEBUG "data": { 05:22:01 DEBUG "resultType": "vector", 05:22:01 DEBUG "result": [ 05:22:01 DEBUG { 05:22:01 DEBUG "metric": { 05:22:01 DEBUG "pod": "ds-idrepo-0" 05:22:01 DEBUG }, 05:22:01 DEBUG "value": [ 05:22:01 DEBUG 05:22:01 DEBUG 05:22:01 DEBUG 05:22:01 DEBUG [...] 05:22:01 DEBUG 05:22:01 DEBUG 05:22:01 DEBUG 05:22:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:22:01 DEBUG }, 05:22:01 DEBUG "value": [ 05:22:01 DEBUG 1735363321, 05:22:01 DEBUG "831074304" 05:22:01 DEBUG ] 05:22:01 DEBUG } 05:22:01 DEBUG ] 05:22:01 DEBUG } 05:22:01 DEBUG } 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363321" 05:22:01 INFO [http_cmd]: http status code OK 05:22:01 DEBUG --- status code --- 05:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:01 DEBUG --- http response --- 05:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:22:01 DEBUG ----- output ----- 05:22:01 DEBUG { 05:22:01 DEBUG "status": "success", 05:22:01 DEBUG "data": { 05:22:01 DEBUG "resultType": "vector", 05:22:01 DEBUG "result": [ 05:22:01 DEBUG { 05:22:01 DEBUG "metric": { 05:22:01 DEBUG "instance": "10.142.0.11:9100" 05:22:01 DEBUG }, 05:22:01 DEBUG "value": [ 05:22:01 DEBUG 05:22:01 DEBUG 05:22:01 DEBUG 05:22:01 DEBUG [...] 05:22:01 DEBUG 05:22:01 DEBUG 05:22:01 DEBUG 05:22:01 DEBUG "instance": "10.142.0.51:9100" 05:22:01 DEBUG }, 05:22:01 DEBUG "value": [ 05:22:01 DEBUG 1735363321, 05:22:01 DEBUG "0.0010333333333278462" 05:22:01 DEBUG ] 05:22:01 DEBUG } 05:22:01 DEBUG ] 05:22:01 DEBUG } 05:22:01 DEBUG } 05:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363321" 05:22:02 INFO [http_cmd]: http status code OK 05:22:02 DEBUG --- status code --- 05:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:02 DEBUG --- http response --- 05:22:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:22:02 DEBUG ----- output ----- 05:22:02 DEBUG { 05:22:02 DEBUG "status": "success", 05:22:02 DEBUG "data": { 05:22:02 DEBUG "resultType": "vector", 05:22:02 DEBUG "result": [ 05:22:02 DEBUG { 05:22:02 DEBUG "metric": { 05:22:02 DEBUG "instance": "10.142.0.11:9100" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG [...] 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG "instance": "10.142.0.51:9100" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 1735363321, 05:22:02 DEBUG "0.0013333333333330683" 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG } 05:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363322" 05:22:02 INFO [http_cmd]: http status code OK 05:22:02 DEBUG --- status code --- 05:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:02 DEBUG --- http response --- 05:22:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:22:02 DEBUG ----- output ----- 05:22:02 DEBUG { 05:22:02 DEBUG "status": "success", 05:22:02 DEBUG "data": { 05:22:02 DEBUG "resultType": "vector", 05:22:02 DEBUG "result": [ 05:22:02 DEBUG { 05:22:02 DEBUG "metric": { 05:22:02 DEBUG "instance": "10.142.0.11:9100" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG [...] 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG "instance": "10.142.0.51:9100" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 1735363322, 05:22:02 DEBUG "1226844.1" 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG } 05:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363322" 05:22:02 INFO [http_cmd]: http status code OK 05:22:02 DEBUG --- status code --- 05:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:02 DEBUG --- http response --- 05:22:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:22:02 DEBUG ----- output ----- 05:22:02 DEBUG { 05:22:02 DEBUG "status": "success", 05:22:02 DEBUG "data": { 05:22:02 DEBUG "resultType": "vector", 05:22:02 DEBUG "result": [ 05:22:02 DEBUG { 05:22:02 DEBUG "metric": { 05:22:02 DEBUG "pod": "ds-cts-0" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG [...] 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG "pod": "ds-idrepo-0" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 1735363322, 05:22:02 DEBUG "2791197.046736784" 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG } 05:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363322" 05:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363322" 05:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363322" 05:22:02 INFO [http_cmd]: http status code OK 05:22:02 DEBUG --- status code --- 05:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:02 DEBUG --- http response --- 05:22:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:22:02 DEBUG ----- output ----- 05:22:02 DEBUG { 05:22:02 DEBUG "status": "success", 05:22:02 DEBUG "data": { 05:22:02 DEBUG "resultType": "vector", 05:22:02 DEBUG "result": [ 05:22:02 DEBUG { 05:22:02 DEBUG "metric": { 05:22:02 DEBUG "pod": "ds-cts-0" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG [...] 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG "pod": "ds-idrepo-0" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 1735363322, 05:22:02 INFO [http_cmd]: http status code OK 05:22:02 DEBUG "0.10650275127070269" 05:22:02 DEBUG --- status code --- 05:22:02 DEBUG ] 05:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:02 DEBUG } 05:22:02 DEBUG --- http response --- 05:22:02 DEBUG ] 05:22:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:22:02 DEBUG } 05:22:02 DEBUG ----- output ----- 05:22:02 DEBUG } 05:22:02 DEBUG { 05:22:02 DEBUG "status": "success", 05:22:02 DEBUG "data": { 05:22:02 DEBUG "resultType": "vector", 05:22:02 DEBUG "result": [ 05:22:02 DEBUG { 05:22:02 DEBUG "metric": { 05:22:02 DEBUG "pod": "ds-cts-0" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG [...] 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG "pod": "ds-idrepo-0" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 1735363322, 05:22:02 DEBUG "427.9445566622554" 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG } 05:22:02 INFO [http_cmd]: http status code OK 05:22:02 DEBUG --- status code --- 05:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:02 DEBUG --- http response --- 05:22:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:22:02 DEBUG ----- output ----- 05:22:02 DEBUG { 05:22:02 DEBUG "status": "success", 05:22:02 DEBUG "data": { 05:22:02 DEBUG "resultType": "vector", 05:22:02 DEBUG "result": [ 05:22:02 DEBUG { 05:22:02 DEBUG "metric": { 05:22:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG [...] 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 1735363322, 05:22:02 DEBUG "14880919552" 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG } 05:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363322" 05:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363322" 05:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363322" 05:22:02 INFO [http_cmd]: http status code OK 05:22:02 DEBUG --- status code --- 05:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:02 DEBUG --- http response --- 05:22:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:22:02 DEBUG ----- output ----- 05:22:02 DEBUG { 05:22:02 DEBUG "status": "success", 05:22:02 DEBUG "data": { 05:22:02 DEBUG "resultType": "vector", 05:22:02 DEBUG "result": [ 05:22:02 DEBUG { 05:22:02 DEBUG "metric": { 05:22:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG [...] 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:22:02 DEBUG }, 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 1735363322, 05:22:02 DEBUG "0.09238174918369213" 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG } 05:22:02 INFO [http_cmd]: http status code OK 05:22:02 DEBUG --- status code --- 05:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:02 INFO [http_cmd]: http status code OK 05:22:02 DEBUG --- http response --- 05:22:02 DEBUG --- status code --- 05:22:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:02 DEBUG ----- output ----- 05:22:02 DEBUG --- http response --- 05:22:02 DEBUG { 05:22:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:22:02 DEBUG "status": "success", 05:22:02 DEBUG ----- output ----- 05:22:02 DEBUG "data": { 05:22:02 DEBUG { 05:22:02 DEBUG "resultType": "vector", 05:22:02 DEBUG "status": "success", 05:22:02 DEBUG "result": [ 05:22:02 DEBUG "data": { 05:22:02 DEBUG { 05:22:02 DEBUG "resultType": "vector", 05:22:02 DEBUG "metric": { 05:22:02 DEBUG "result": [ 05:22:02 DEBUG "instance": "10.142.0.11:9100" 05:22:02 DEBUG { 05:22:02 DEBUG }, 05:22:02 DEBUG "metric": { 05:22:02 DEBUG "value": [ 05:22:02 DEBUG "instance": "10.142.0.11:9100" 05:22:02 DEBUG 05:22:02 DEBUG }, 05:22:02 DEBUG 05:22:02 DEBUG "value": [ 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG [...] 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG [...] 05:22:02 DEBUG 05:22:02 DEBUG 05:22:02 DEBUG "instance": "10.142.0.51:9100" 05:22:02 DEBUG 05:22:02 DEBUG }, 05:22:02 DEBUG 05:22:02 DEBUG "value": [ 05:22:02 DEBUG "instance": "10.142.0.51:9100" 05:22:02 DEBUG 1735363322, 05:22:02 DEBUG }, 05:22:02 DEBUG "1245899.533333334" 05:22:02 DEBUG "value": [ 05:22:02 DEBUG ] 05:22:02 DEBUG 1735363322, 05:22:02 DEBUG } 05:22:02 DEBUG "16793.600000000002" 05:22:02 DEBUG ] 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG } 05:22:02 DEBUG } 05:22:02 DEBUG ] 05:22:02 DEBUG } 05:22:02 DEBUG } 05:22:32 INFO 05:22:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:22:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:22:32 INFO [loop_until]: OK (rc = 0) 05:22:32 DEBUG --- stdout --- 05:22:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 533Mi ds-idrepo-0 67m 13378Mi rcs-687d95c5b-qs2xv 83m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:22:32 DEBUG --- stderr --- 05:22:32 DEBUG 05:22:34 INFO 05:22:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:22:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:22:34 INFO [loop_until]: OK (rc = 0) 05:22:34 DEBUG --- stdout --- 05:22:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 167m 1% 3452Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 140m 0% 18872Mi 66% 05:22:34 DEBUG --- stderr --- 05:22:34 DEBUG 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG --- http response --- 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG --- status code --- 05:23:01 ERROR [http_cmd]: ERROR 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 ERROR --- status code --- 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG --- http response --- 05:23:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 ERROR --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG --- status code --- 05:23:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363381" 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:23:01 DEBUG ----- output ----- 05:23:01 DEBUG { 05:23:01 DEBUG "status": "success", 05:23:01 DEBUG "data": { 05:23:01 DEBUG "resultType": "vector", 05:23:01 DEBUG "result": [ 05:23:01 DEBUG { 05:23:01 DEBUG "metric": { 05:23:01 DEBUG "pod": "ds-idrepo-0" 05:23:01 DEBUG }, 05:23:01 DEBUG "value": [ 05:23:01 DEBUG 05:23:01 DEBUG 05:23:01 DEBUG 05:23:01 DEBUG [...] 05:23:01 DEBUG 05:23:01 DEBUG 05:23:01 DEBUG 05:23:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:23:01 DEBUG }, 05:23:01 DEBUG "value": [ 05:23:01 DEBUG 1735363381, 05:23:01 DEBUG "831074304" 05:23:01 DEBUG ] 05:23:01 DEBUG } 05:23:01 DEBUG ] 05:23:01 DEBUG } 05:23:01 DEBUG } 05:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363381" 05:23:01 INFO [http_cmd]: http status code OK 05:23:01 DEBUG --- status code --- 05:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:01 DEBUG --- http response --- 05:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:23:01 DEBUG ----- output ----- 05:23:01 DEBUG { 05:23:01 DEBUG "status": "success", 05:23:01 DEBUG "data": { 05:23:01 DEBUG "resultType": "vector", 05:23:01 DEBUG "result": [ 05:23:01 DEBUG { 05:23:01 DEBUG "metric": { 05:23:01 DEBUG "instance": "10.142.0.11:9100" 05:23:01 DEBUG }, 05:23:01 DEBUG "value": [ 05:23:01 DEBUG 05:23:01 DEBUG 05:23:01 DEBUG 05:23:01 DEBUG [...] 05:23:01 DEBUG 05:23:01 DEBUG 05:23:01 DEBUG 05:23:01 DEBUG "instance": "10.142.0.51:9100" 05:23:01 DEBUG }, 05:23:01 DEBUG "value": [ 05:23:01 DEBUG 1735363381, 05:23:01 DEBUG "0.0010333333333354252" 05:23:01 DEBUG ] 05:23:01 DEBUG } 05:23:01 DEBUG ] 05:23:01 DEBUG } 05:23:01 DEBUG } 05:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363382" 05:23:02 INFO [http_cmd]: http status code OK 05:23:02 DEBUG --- status code --- 05:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:02 DEBUG --- http response --- 05:23:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:23:02 DEBUG ----- output ----- 05:23:02 DEBUG { 05:23:02 DEBUG "status": "success", 05:23:02 DEBUG "data": { 05:23:02 DEBUG "resultType": "vector", 05:23:02 DEBUG "result": [ 05:23:02 DEBUG { 05:23:02 DEBUG "metric": { 05:23:02 DEBUG "instance": "10.142.0.11:9100" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG [...] 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG "instance": "10.142.0.51:9100" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 1735363382, 05:23:02 DEBUG "0.0006666666666665342" 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG } 05:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363382" 05:23:02 INFO [http_cmd]: http status code OK 05:23:02 DEBUG --- status code --- 05:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:02 DEBUG --- http response --- 05:23:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:23:02 DEBUG ----- output ----- 05:23:02 DEBUG { 05:23:02 DEBUG "status": "success", 05:23:02 DEBUG "data": { 05:23:02 DEBUG "resultType": "vector", 05:23:02 DEBUG "result": [ 05:23:02 DEBUG { 05:23:02 DEBUG "metric": { 05:23:02 DEBUG "instance": "10.142.0.11:9100" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG [...] 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG "instance": "10.142.0.51:9100" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 1735363382, 05:23:02 DEBUG "1223915.7333333334" 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG } 05:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363382" 05:23:02 INFO [http_cmd]: http status code OK 05:23:02 DEBUG --- status code --- 05:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:02 DEBUG --- http response --- 05:23:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:23:02 DEBUG ----- output ----- 05:23:02 DEBUG { 05:23:02 DEBUG "status": "success", 05:23:02 DEBUG "data": { 05:23:02 DEBUG "resultType": "vector", 05:23:02 DEBUG "result": [ 05:23:02 DEBUG { 05:23:02 DEBUG "metric": { 05:23:02 DEBUG "pod": "ds-idrepo-0" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG [...] 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 1735363382, 05:23:02 DEBUG "0" 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG } 05:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363382" 05:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363382" 05:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363382" 05:23:02 INFO [http_cmd]: http status code OK 05:23:02 DEBUG --- status code --- 05:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:02 DEBUG --- http response --- 05:23:02 INFO [http_cmd]: http status code OK 05:23:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:23:02 DEBUG --- status code --- 05:23:02 DEBUG ----- output ----- 05:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:02 DEBUG { 05:23:02 DEBUG --- http response --- 05:23:02 DEBUG "status": "success", 05:23:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:23:02 DEBUG "data": { 05:23:02 DEBUG ----- output ----- 05:23:02 DEBUG "resultType": "vector", 05:23:02 DEBUG { 05:23:02 DEBUG "result": [ 05:23:02 DEBUG "status": "success", 05:23:02 DEBUG { 05:23:02 DEBUG "data": { 05:23:02 DEBUG "metric": { 05:23:02 DEBUG "resultType": "vector", 05:23:02 DEBUG "pod": "ds-idrepo-0" 05:23:02 DEBUG "result": [ 05:23:02 DEBUG }, 05:23:02 DEBUG { 05:23:02 DEBUG "value": [ 05:23:02 DEBUG "metric": { 05:23:02 DEBUG 05:23:02 DEBUG "pod": "ds-idrepo-0" 05:23:02 DEBUG 05:23:02 DEBUG }, 05:23:02 DEBUG 05:23:02 DEBUG "value": [ 05:23:02 DEBUG [...] 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG [...] 05:23:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:23:02 DEBUG 05:23:02 DEBUG }, 05:23:02 DEBUG 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 05:23:02 DEBUG 1735363382, 05:23:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:23:02 DEBUG "0.0012511179210490278" 05:23:02 DEBUG }, 05:23:02 DEBUG ] 05:23:02 DEBUG "value": [ 05:23:02 DEBUG } 05:23:02 DEBUG 1735363382, 05:23:02 DEBUG ] 05:23:02 DEBUG "727.540882385508" 05:23:02 DEBUG } 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG } 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG } 05:23:02 INFO [http_cmd]: http status code OK 05:23:02 DEBUG --- status code --- 05:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:02 DEBUG --- http response --- 05:23:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:23:02 DEBUG ----- output ----- 05:23:02 DEBUG { 05:23:02 DEBUG "status": "success", 05:23:02 DEBUG "data": { 05:23:02 DEBUG "resultType": "vector", 05:23:02 DEBUG "result": [ 05:23:02 DEBUG { 05:23:02 DEBUG "metric": { 05:23:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG [...] 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 1735363382, 05:23:02 DEBUG "14882095104" 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG } 05:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363382" 05:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363382" 05:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363382" 05:23:02 INFO [http_cmd]: http status code OK 05:23:02 DEBUG --- status code --- 05:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:02 DEBUG --- http response --- 05:23:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:23:02 DEBUG ----- output ----- 05:23:02 DEBUG { 05:23:02 DEBUG "status": "success", 05:23:02 DEBUG "data": { 05:23:02 DEBUG "resultType": "vector", 05:23:02 DEBUG "result": [ 05:23:02 DEBUG { 05:23:02 DEBUG "metric": { 05:23:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG [...] 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 1735363382, 05:23:02 DEBUG "0.0978774690100748" 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG } 05:23:02 INFO [http_cmd]: http status code OK 05:23:02 DEBUG --- status code --- 05:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:02 DEBUG --- http response --- 05:23:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:23:02 DEBUG ----- output ----- 05:23:02 DEBUG { 05:23:02 DEBUG "status": "success", 05:23:02 DEBUG "data": { 05:23:02 DEBUG "resultType": "vector", 05:23:02 DEBUG "result": [ 05:23:02 DEBUG { 05:23:02 DEBUG "metric": { 05:23:02 DEBUG "instance": "10.142.0.11:9100" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG [...] 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG "instance": "10.142.0.51:9100" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 1735363382, 05:23:02 DEBUG "1242935.666666667" 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG } 05:23:02 INFO [http_cmd]: http status code OK 05:23:02 DEBUG --- status code --- 05:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:02 DEBUG --- http response --- 05:23:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:23:02 DEBUG ----- output ----- 05:23:02 DEBUG { 05:23:02 DEBUG "status": "success", 05:23:02 DEBUG "data": { 05:23:02 DEBUG "resultType": "vector", 05:23:02 DEBUG "result": [ 05:23:02 DEBUG { 05:23:02 DEBUG "metric": { 05:23:02 DEBUG "instance": "10.142.0.11:9100" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG [...] 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG 05:23:02 DEBUG "instance": "10.142.0.51:9100" 05:23:02 DEBUG }, 05:23:02 DEBUG "value": [ 05:23:02 DEBUG 1735363382, 05:23:02 DEBUG "23893.333333333336" 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG ] 05:23:02 DEBUG } 05:23:02 DEBUG } 05:23:32 INFO 05:23:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:23:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:23:32 INFO [loop_until]: OK (rc = 0) 05:23:32 DEBUG --- stdout --- 05:23:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 602m 13374Mi rcs-687d95c5b-qs2xv 56m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:23:32 DEBUG --- stderr --- 05:23:32 DEBUG 05:23:34 INFO 05:23:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:23:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:23:34 INFO [loop_until]: OK (rc = 0) 05:23:34 DEBUG --- stdout --- 05:23:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 129m 0% 3457Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 262m 0% 18875Mi 66% 05:23:34 DEBUG --- stderr --- 05:23:34 DEBUG 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 ERROR [http_cmd]: ERROR 05:24:01 ERROR --- status code --- 05:24:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:24:01 ERROR --- http response --- 05:24:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:24:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363441" 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:24:01 DEBUG ----- output ----- 05:24:01 DEBUG { 05:24:01 DEBUG "status": "success", 05:24:01 DEBUG "data": { 05:24:01 DEBUG "resultType": "vector", 05:24:01 DEBUG "result": [ 05:24:01 DEBUG { 05:24:01 DEBUG "metric": { 05:24:01 DEBUG "pod": "ds-idrepo-0" 05:24:01 DEBUG }, 05:24:01 DEBUG "value": [ 05:24:01 DEBUG 05:24:01 DEBUG 05:24:01 DEBUG 05:24:01 DEBUG [...] 05:24:01 DEBUG 05:24:01 DEBUG 05:24:01 DEBUG 05:24:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:24:01 DEBUG }, 05:24:01 DEBUG "value": [ 05:24:01 DEBUG 1735363441, 05:24:01 DEBUG "831074304" 05:24:01 DEBUG ] 05:24:01 DEBUG } 05:24:01 DEBUG ] 05:24:01 DEBUG } 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG } 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363441" 05:24:01 INFO [http_cmd]: http status code OK 05:24:01 DEBUG --- status code --- 05:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:01 DEBUG --- http response --- 05:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:24:01 DEBUG ----- output ----- 05:24:01 DEBUG { 05:24:01 DEBUG "status": "success", 05:24:01 DEBUG "data": { 05:24:01 DEBUG "resultType": "vector", 05:24:01 DEBUG "result": [ 05:24:01 DEBUG { 05:24:01 DEBUG "metric": { 05:24:01 DEBUG "instance": "10.142.0.11:9100" 05:24:01 DEBUG }, 05:24:01 DEBUG "value": [ 05:24:01 DEBUG 05:24:01 DEBUG 05:24:01 DEBUG 05:24:01 DEBUG [...] 05:24:01 DEBUG 05:24:01 DEBUG 05:24:01 DEBUG 05:24:01 DEBUG "instance": "10.142.0.51:9100" 05:24:01 DEBUG }, 05:24:01 DEBUG "value": [ 05:24:01 DEBUG 1735363441, 05:24:01 DEBUG "0.000400000000006647" 05:24:01 DEBUG ] 05:24:01 DEBUG } 05:24:01 DEBUG ] 05:24:01 DEBUG } 05:24:01 DEBUG } 05:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363442" 05:24:02 INFO [http_cmd]: http status code OK 05:24:02 DEBUG --- status code --- 05:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:02 DEBUG --- http response --- 05:24:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:24:02 DEBUG ----- output ----- 05:24:02 DEBUG { 05:24:02 DEBUG "status": "success", 05:24:02 DEBUG "data": { 05:24:02 DEBUG "resultType": "vector", 05:24:02 DEBUG "result": [ 05:24:02 DEBUG { 05:24:02 DEBUG "metric": { 05:24:02 DEBUG "instance": "10.142.0.11:9100" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG [...] 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG "instance": "10.142.0.51:9100" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 1735363442, 05:24:02 DEBUG "0.0006666666666670078" 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG } 05:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363442" 05:24:02 INFO [http_cmd]: http status code OK 05:24:02 DEBUG --- status code --- 05:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:02 DEBUG --- http response --- 05:24:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:24:02 DEBUG ----- output ----- 05:24:02 DEBUG { 05:24:02 DEBUG "status": "success", 05:24:02 DEBUG "data": { 05:24:02 DEBUG "resultType": "vector", 05:24:02 DEBUG "result": [ 05:24:02 DEBUG { 05:24:02 DEBUG "metric": { 05:24:02 DEBUG "instance": "10.142.0.11:9100" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG [...] 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG "instance": "10.142.0.51:9100" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 1735363442, 05:24:02 DEBUG "73673.3" 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG } 05:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363442" 05:24:02 INFO [http_cmd]: http status code OK 05:24:02 DEBUG --- status code --- 05:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:02 DEBUG --- http response --- 05:24:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:24:02 DEBUG ----- output ----- 05:24:02 DEBUG { 05:24:02 DEBUG "status": "success", 05:24:02 DEBUG "data": { 05:24:02 DEBUG "resultType": "vector", 05:24:02 DEBUG "result": [ 05:24:02 DEBUG { 05:24:02 DEBUG "metric": { 05:24:02 DEBUG "pod": "ds-idrepo-0" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG [...] 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 1735363442, 05:24:02 DEBUG "0" 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG } 05:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363442" 05:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363442" 05:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363442" 05:24:02 INFO [http_cmd]: http status code OK 05:24:02 DEBUG --- status code --- 05:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:02 DEBUG --- http response --- 05:24:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:24:02 DEBUG ----- output ----- 05:24:02 DEBUG { 05:24:02 DEBUG "status": "success", 05:24:02 DEBUG "data": { 05:24:02 DEBUG "resultType": "vector", 05:24:02 DEBUG "result": [ 05:24:02 DEBUG { 05:24:02 DEBUG "metric": { 05:24:02 DEBUG "pod": "ds-idrepo-0" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG [...] 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 1735363442, 05:24:02 DEBUG "421.37027638707906" 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG } 05:24:02 INFO [http_cmd]: http status code OK 05:24:02 DEBUG --- status code --- 05:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:02 DEBUG --- http response --- 05:24:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:24:02 DEBUG ----- output ----- 05:24:02 DEBUG { 05:24:02 DEBUG "status": "success", 05:24:02 DEBUG "data": { 05:24:02 DEBUG "resultType": "vector", 05:24:02 DEBUG "result": [ 05:24:02 DEBUG { 05:24:02 DEBUG "metric": { 05:24:02 DEBUG "pod": "ds-idrepo-0" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG [...] 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 1735363442, 05:24:02 DEBUG "0.001257256295649458" 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG } 05:24:02 INFO [http_cmd]: http status code OK 05:24:02 DEBUG --- status code --- 05:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:02 DEBUG --- http response --- 05:24:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:24:02 DEBUG ----- output ----- 05:24:02 DEBUG { 05:24:02 DEBUG "status": "success", 05:24:02 DEBUG "data": { 05:24:02 DEBUG "resultType": "vector", 05:24:02 DEBUG "result": [ 05:24:02 DEBUG { 05:24:02 DEBUG "metric": { 05:24:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG [...] 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 1735363442, 05:24:02 DEBUG "14879277056" 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG } 05:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363442" 05:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363442" 05:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363442" 05:24:02 INFO [http_cmd]: http status code OK 05:24:02 DEBUG --- status code --- 05:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:02 DEBUG --- http response --- 05:24:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:24:02 DEBUG ----- output ----- 05:24:02 DEBUG { 05:24:02 INFO [http_cmd]: http status code OK 05:24:02 DEBUG "status": "success", 05:24:02 INFO [http_cmd]: http status code OK 05:24:02 DEBUG --- status code --- 05:24:02 DEBUG "data": { 05:24:02 DEBUG --- status code --- 05:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:02 DEBUG "resultType": "vector", 05:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:02 DEBUG --- http response --- 05:24:02 DEBUG "result": [ 05:24:02 DEBUG --- http response --- 05:24:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:24:02 DEBUG ----- output ----- 05:24:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:24:02 DEBUG { 05:24:02 DEBUG { 05:24:02 DEBUG ----- output ----- 05:24:02 DEBUG "metric": { 05:24:02 DEBUG "status": "success", 05:24:02 DEBUG { 05:24:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:24:02 DEBUG "data": { 05:24:02 DEBUG "status": "success", 05:24:02 DEBUG }, 05:24:02 DEBUG "resultType": "vector", 05:24:02 DEBUG "result": [ 05:24:02 DEBUG "value": [ 05:24:02 DEBUG "data": { 05:24:02 DEBUG { 05:24:02 DEBUG "metric": { 05:24:02 DEBUG "resultType": "vector", 05:24:02 DEBUG 05:24:02 DEBUG "instance": "10.142.0.11:9100" 05:24:02 DEBUG }, 05:24:02 DEBUG 05:24:02 DEBUG "result": [ 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 05:24:02 DEBUG { 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG [...] 05:24:02 DEBUG "metric": { 05:24:02 DEBUG [...] 05:24:02 DEBUG 05:24:02 DEBUG "instance": "10.142.0.11:9100" 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG 05:24:02 DEBUG }, 05:24:02 DEBUG "instance": "10.142.0.51:9100" 05:24:02 DEBUG }, 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 1735363442, 05:24:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:24:02 DEBUG 05:24:02 DEBUG "92696.70000000001" 05:24:02 DEBUG }, 05:24:02 DEBUG 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG 05:24:02 DEBUG "value": [ 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG 1735363442, 05:24:02 DEBUG [...] 05:24:02 DEBUG } 05:24:02 DEBUG "0.1424658641529033" 05:24:02 DEBUG 05:24:02 DEBUG ] 05:24:02 DEBUG 05:24:02 DEBUG } 05:24:02 DEBUG 05:24:02 DEBUG ] 05:24:02 DEBUG "instance": "10.142.0.51:9100" 05:24:02 DEBUG } 05:24:02 DEBUG }, 05:24:02 DEBUG } 05:24:02 DEBUG "value": [ 05:24:02 DEBUG 1735363442, 05:24:02 DEBUG "26760.533333333336" 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG ] 05:24:02 DEBUG } 05:24:02 DEBUG } 05:24:32 INFO 05:24:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:24:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:24:32 INFO [loop_until]: OK (rc = 0) 05:24:32 DEBUG --- stdout --- 05:24:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 23m 13375Mi rcs-687d95c5b-qs2xv 2m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:24:32 DEBUG --- stderr --- 05:24:32 DEBUG 05:24:34 INFO 05:24:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:24:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:24:34 INFO [loop_until]: OK (rc = 0) 05:24:34 DEBUG --- stdout --- 05:24:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3448Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 99m 0% 18868Mi 66% 05:24:34 DEBUG --- stderr --- 05:24:34 DEBUG 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG --- status code --- 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 ERROR [http_cmd]: ERROR 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 ERROR --- status code --- 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG --- status code --- 05:25:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 ERROR --- http response --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG --- http response --- 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363501" 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG --- status code --- 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG --- status code --- 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:25:01 DEBUG ----- output ----- 05:25:01 DEBUG { 05:25:01 DEBUG "status": "success", 05:25:01 DEBUG "data": { 05:25:01 DEBUG "resultType": "vector", 05:25:01 DEBUG "result": [ 05:25:01 DEBUG { 05:25:01 DEBUG "metric": { 05:25:01 DEBUG "pod": "ds-idrepo-0" 05:25:01 DEBUG }, 05:25:01 DEBUG "value": [ 05:25:01 DEBUG 05:25:01 DEBUG 05:25:01 DEBUG 05:25:01 DEBUG [...] 05:25:01 DEBUG 05:25:01 DEBUG 05:25:01 DEBUG 05:25:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:25:01 DEBUG }, 05:25:01 DEBUG "value": [ 05:25:01 DEBUG 1735363501, 05:25:01 DEBUG "831074304" 05:25:01 DEBUG ] 05:25:01 DEBUG } 05:25:01 DEBUG ] 05:25:01 DEBUG } 05:25:01 DEBUG } 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363501" 05:25:01 INFO [http_cmd]: http status code OK 05:25:01 DEBUG --- status code --- 05:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:01 DEBUG --- http response --- 05:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:25:01 DEBUG ----- output ----- 05:25:01 DEBUG { 05:25:01 DEBUG "status": "success", 05:25:01 DEBUG "data": { 05:25:01 DEBUG "resultType": "vector", 05:25:01 DEBUG "result": [ 05:25:01 DEBUG { 05:25:01 DEBUG "metric": { 05:25:01 DEBUG "instance": "10.142.0.11:9100" 05:25:01 DEBUG }, 05:25:01 DEBUG "value": [ 05:25:01 DEBUG 05:25:01 DEBUG 05:25:01 DEBUG 05:25:01 DEBUG [...] 05:25:01 DEBUG 05:25:01 DEBUG 05:25:01 DEBUG 05:25:01 DEBUG "instance": "10.142.0.51:9100" 05:25:01 DEBUG }, 05:25:01 DEBUG "value": [ 05:25:01 DEBUG 1735363501, 05:25:01 DEBUG "0.000666666666662271" 05:25:01 DEBUG ] 05:25:01 DEBUG } 05:25:01 DEBUG ] 05:25:01 DEBUG } 05:25:01 DEBUG } 05:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363502" 05:25:02 INFO [http_cmd]: http status code OK 05:25:02 DEBUG --- status code --- 05:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:02 DEBUG --- http response --- 05:25:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:25:02 DEBUG ----- output ----- 05:25:02 DEBUG { 05:25:02 DEBUG "status": "success", 05:25:02 DEBUG "data": { 05:25:02 DEBUG "resultType": "vector", 05:25:02 DEBUG "result": [ 05:25:02 DEBUG { 05:25:02 DEBUG "metric": { 05:25:02 DEBUG "instance": "10.142.0.11:9100" 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG [...] 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG "instance": "10.142.0.51:9100" 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 1735363502, 05:25:02 DEBUG "0.0003333333333335039" 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG } 05:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363502" 05:25:02 INFO [http_cmd]: http status code OK 05:25:02 DEBUG --- status code --- 05:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:02 DEBUG --- http response --- 05:25:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:25:02 DEBUG ----- output ----- 05:25:02 DEBUG { 05:25:02 DEBUG "status": "success", 05:25:02 DEBUG "data": { 05:25:02 DEBUG "resultType": "vector", 05:25:02 DEBUG "result": [ 05:25:02 DEBUG { 05:25:02 DEBUG "metric": { 05:25:02 DEBUG "instance": "10.142.0.11:9100" 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG [...] 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG "instance": "10.142.0.51:9100" 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 1735363502, 05:25:02 DEBUG "73858.60000000002" 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG } 05:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363502" 05:25:02 INFO [http_cmd]: http status code OK 05:25:02 DEBUG --- status code --- 05:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:02 DEBUG --- http response --- 05:25:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:25:02 DEBUG ----- output ----- 05:25:02 DEBUG { 05:25:02 DEBUG "status": "success", 05:25:02 DEBUG "data": { 05:25:02 DEBUG "resultType": "vector", 05:25:02 DEBUG "result": [ 05:25:02 DEBUG { 05:25:02 DEBUG "metric": { 05:25:02 DEBUG "pod": "ds-idrepo-0" 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG [...] 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 1735363502, 05:25:02 DEBUG "0" 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG } 05:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363502" 05:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363502" 05:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363502" 05:25:02 INFO [http_cmd]: http status code OK 05:25:02 DEBUG --- status code --- 05:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:02 DEBUG --- http response --- 05:25:02 INFO [http_cmd]: http status code OK 05:25:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:25:02 DEBUG --- status code --- 05:25:02 DEBUG ----- output ----- 05:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:02 DEBUG { 05:25:02 DEBUG --- http response --- 05:25:02 DEBUG "status": "success", 05:25:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:25:02 DEBUG "data": { 05:25:02 DEBUG ----- output ----- 05:25:02 DEBUG "resultType": "vector", 05:25:02 DEBUG { 05:25:02 DEBUG "result": [ 05:25:02 DEBUG "status": "success", 05:25:02 DEBUG { 05:25:02 DEBUG "data": { 05:25:02 DEBUG "metric": { 05:25:02 INFO [http_cmd]: http status code OK 05:25:02 DEBUG "resultType": "vector", 05:25:02 DEBUG "pod": "ds-idrepo-0" 05:25:02 DEBUG --- status code --- 05:25:02 DEBUG "result": [ 05:25:02 DEBUG }, 05:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:02 DEBUG { 05:25:02 DEBUG "value": [ 05:25:02 DEBUG --- http response --- 05:25:02 DEBUG "metric": { 05:25:02 DEBUG 05:25:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:25:02 DEBUG "pod": "ds-idrepo-0" 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG }, 05:25:02 DEBUG ----- output ----- 05:25:02 DEBUG [...] 05:25:02 DEBUG "value": [ 05:25:02 DEBUG { 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG "status": "success", 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG "data": { 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG "resultType": "vector", 05:25:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:25:02 DEBUG [...] 05:25:02 DEBUG "result": [ 05:25:02 DEBUG }, 05:25:02 DEBUG 05:25:02 DEBUG { 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 05:25:02 DEBUG "metric": { 05:25:02 DEBUG 1735363502, 05:25:02 DEBUG 05:25:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:25:02 DEBUG "478.4934931426836" 05:25:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:25:02 DEBUG }, 05:25:02 DEBUG ] 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG } 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 05:25:02 DEBUG ] 05:25:02 DEBUG 1735363502, 05:25:02 DEBUG 05:25:02 DEBUG } 05:25:02 DEBUG "0.0012843355870581978" 05:25:02 DEBUG 05:25:02 DEBUG } 05:25:02 DEBUG ] 05:25:02 DEBUG [...] 05:25:02 DEBUG } 05:25:02 DEBUG 05:25:02 DEBUG ] 05:25:02 DEBUG 05:25:02 DEBUG } 05:25:02 DEBUG 05:25:02 DEBUG } 05:25:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 1735363502, 05:25:02 DEBUG "14879981568" 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG } 05:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363502" 05:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363502" 05:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363502" 05:25:02 INFO [http_cmd]: http status code OK 05:25:02 DEBUG --- status code --- 05:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:02 DEBUG --- http response --- 05:25:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:25:02 DEBUG ----- output ----- 05:25:02 DEBUG { 05:25:02 DEBUG "status": "success", 05:25:02 DEBUG "data": { 05:25:02 DEBUG "resultType": "vector", 05:25:02 DEBUG "result": [ 05:25:02 DEBUG { 05:25:02 DEBUG "metric": { 05:25:02 DEBUG "instance": "10.142.0.11:9100" 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG [...] 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG "instance": "10.142.0.51:9100" 05:25:02 DEBUG }, 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 1735363502, 05:25:02 DEBUG "93481.93333333332" 05:25:02 DEBUG ] 05:25:02 INFO [http_cmd]: http status code OK 05:25:02 DEBUG } 05:25:02 DEBUG --- status code --- 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG } 05:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:02 INFO [http_cmd]: http status code OK 05:25:02 DEBUG --- http response --- 05:25:02 DEBUG --- status code --- 05:25:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:02 DEBUG ----- output ----- 05:25:02 DEBUG --- http response --- 05:25:02 DEBUG { 05:25:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:25:02 DEBUG "status": "success", 05:25:02 DEBUG ----- output ----- 05:25:02 DEBUG "data": { 05:25:02 DEBUG { 05:25:02 DEBUG "resultType": "vector", 05:25:02 DEBUG "status": "success", 05:25:02 DEBUG "result": [ 05:25:02 DEBUG "data": { 05:25:02 DEBUG { 05:25:02 DEBUG "resultType": "vector", 05:25:02 DEBUG "metric": { 05:25:02 DEBUG "result": [ 05:25:02 DEBUG "instance": "10.142.0.11:9100" 05:25:02 DEBUG { 05:25:02 DEBUG }, 05:25:02 DEBUG "metric": { 05:25:02 DEBUG "value": [ 05:25:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:25:02 DEBUG 05:25:02 DEBUG }, 05:25:02 DEBUG 05:25:02 DEBUG "value": [ 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG [...] 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG [...] 05:25:02 DEBUG 05:25:02 DEBUG 05:25:02 DEBUG "instance": "10.142.0.51:9100" 05:25:02 DEBUG 05:25:02 DEBUG }, 05:25:02 DEBUG 05:25:02 DEBUG "value": [ 05:25:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:25:02 DEBUG 1735363502, 05:25:02 DEBUG }, 05:25:02 DEBUG "26487.466666666674" 05:25:02 DEBUG "value": [ 05:25:02 DEBUG ] 05:25:02 DEBUG 1735363502, 05:25:02 DEBUG } 05:25:02 DEBUG "0.08287625467401621" 05:25:02 DEBUG ] 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG } 05:25:02 DEBUG } 05:25:02 DEBUG ] 05:25:02 DEBUG } 05:25:02 DEBUG } 05:25:32 INFO 05:25:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:25:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:25:32 INFO [loop_until]: OK (rc = 0) 05:25:32 DEBUG --- stdout --- 05:25:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13375Mi rcs-687d95c5b-qs2xv 28m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:25:32 DEBUG --- stderr --- 05:25:32 DEBUG 05:25:34 INFO 05:25:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:25:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:25:34 INFO [loop_until]: OK (rc = 0) 05:25:34 DEBUG --- stdout --- 05:25:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 135m 0% 3422Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 99m 0% 18873Mi 66% 05:25:34 DEBUG --- stderr --- 05:25:34 DEBUG 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG --- status code --- 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363561" 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG --- status code --- 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 ERROR [http_cmd]: ERROR 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG --- status code --- 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 ERROR --- status code --- 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG --- http response --- 05:26:01 ERROR --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363561" 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:26:01 DEBUG ----- output ----- 05:26:01 DEBUG { 05:26:01 DEBUG "status": "success", 05:26:01 DEBUG "data": { 05:26:01 DEBUG "resultType": "vector", 05:26:01 DEBUG "result": [ 05:26:01 DEBUG { 05:26:01 DEBUG "metric": { 05:26:01 DEBUG "pod": "ds-idrepo-0" 05:26:01 DEBUG }, 05:26:01 DEBUG "value": [ 05:26:01 DEBUG 05:26:01 DEBUG 05:26:01 DEBUG 05:26:01 DEBUG [...] 05:26:01 DEBUG 05:26:01 DEBUG 05:26:01 DEBUG 05:26:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:26:01 DEBUG }, 05:26:01 DEBUG "value": [ 05:26:01 DEBUG 1735363561, 05:26:01 DEBUG "831303680" 05:26:01 DEBUG ] 05:26:01 DEBUG } 05:26:01 DEBUG ] 05:26:01 DEBUG } 05:26:01 DEBUG } 05:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363561" 05:26:01 INFO [http_cmd]: http status code OK 05:26:01 DEBUG --- status code --- 05:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:01 DEBUG --- http response --- 05:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:26:01 DEBUG ----- output ----- 05:26:01 DEBUG { 05:26:01 DEBUG "status": "success", 05:26:01 DEBUG "data": { 05:26:01 DEBUG "resultType": "vector", 05:26:01 DEBUG "result": [ 05:26:01 DEBUG { 05:26:01 DEBUG "metric": { 05:26:01 DEBUG "instance": "10.142.0.11:9100" 05:26:01 DEBUG }, 05:26:01 DEBUG "value": [ 05:26:01 DEBUG 05:26:01 DEBUG 05:26:01 DEBUG 05:26:01 DEBUG [...] 05:26:01 DEBUG 05:26:01 DEBUG 05:26:01 DEBUG 05:26:01 DEBUG "instance": "10.142.0.51:9100" 05:26:01 DEBUG }, 05:26:01 DEBUG "value": [ 05:26:01 DEBUG 1735363561, 05:26:01 DEBUG "0.00036666666666557537" 05:26:01 DEBUG ] 05:26:01 DEBUG } 05:26:01 DEBUG ] 05:26:01 DEBUG } 05:26:01 DEBUG } 05:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363562" 05:26:02 INFO [http_cmd]: http status code OK 05:26:02 DEBUG --- status code --- 05:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:02 DEBUG --- http response --- 05:26:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:26:02 DEBUG ----- output ----- 05:26:02 DEBUG { 05:26:02 DEBUG "status": "success", 05:26:02 DEBUG "data": { 05:26:02 DEBUG "resultType": "vector", 05:26:02 DEBUG "result": [ 05:26:02 DEBUG { 05:26:02 DEBUG "metric": { 05:26:02 DEBUG "instance": "10.142.0.11:9100" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG [...] 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG "instance": "10.142.0.51:9100" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 1735363562, 05:26:02 DEBUG "0.0006666666666665342" 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363562" 05:26:02 INFO [http_cmd]: http status code OK 05:26:02 DEBUG --- status code --- 05:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:02 DEBUG --- http response --- 05:26:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:26:02 DEBUG ----- output ----- 05:26:02 DEBUG { 05:26:02 DEBUG "status": "success", 05:26:02 DEBUG "data": { 05:26:02 DEBUG "resultType": "vector", 05:26:02 DEBUG "result": [ 05:26:02 DEBUG { 05:26:02 DEBUG "metric": { 05:26:02 DEBUG "instance": "10.142.0.11:9100" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG [...] 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG "instance": "10.142.0.51:9100" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 1735363562, 05:26:02 DEBUG "261457.60000000003" 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363562" 05:26:02 INFO [http_cmd]: http status code OK 05:26:02 DEBUG --- status code --- 05:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:02 DEBUG --- http response --- 05:26:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:26:02 DEBUG ----- output ----- 05:26:02 DEBUG { 05:26:02 DEBUG "status": "success", 05:26:02 DEBUG "data": { 05:26:02 DEBUG "resultType": "vector", 05:26:02 DEBUG "result": [ 05:26:02 DEBUG { 05:26:02 DEBUG "metric": { 05:26:02 DEBUG "pod": "ds-idrepo-0" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG [...] 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 1735363562, 05:26:02 DEBUG "0" 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363562" 05:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363562" 05:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363562" 05:26:02 INFO [http_cmd]: http status code OK 05:26:02 DEBUG --- status code --- 05:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:02 DEBUG --- http response --- 05:26:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:26:02 DEBUG ----- output ----- 05:26:02 DEBUG { 05:26:02 DEBUG "status": "success", 05:26:02 DEBUG "data": { 05:26:02 DEBUG "resultType": "vector", 05:26:02 DEBUG "result": [ 05:26:02 DEBUG { 05:26:02 DEBUG "metric": { 05:26:02 DEBUG "pod": "ds-idrepo-0" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG [...] 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 1735363562, 05:26:02 DEBUG "1255.6859458630502" 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 INFO [http_cmd]: http status code OK 05:26:02 DEBUG --- status code --- 05:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:02 DEBUG --- http response --- 05:26:02 INFO [http_cmd]: http status code OK 05:26:02 DEBUG --- status code --- 05:26:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:02 DEBUG ----- output ----- 05:26:02 DEBUG --- http response --- 05:26:02 DEBUG { 05:26:02 DEBUG "status": "success", 05:26:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:26:02 DEBUG "data": { 05:26:02 DEBUG ----- output ----- 05:26:02 DEBUG "resultType": "vector", 05:26:02 DEBUG { 05:26:02 DEBUG "result": [ 05:26:02 DEBUG "status": "success", 05:26:02 DEBUG { 05:26:02 DEBUG "data": { 05:26:02 DEBUG "metric": { 05:26:02 DEBUG "resultType": "vector", 05:26:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:26:02 DEBUG "result": [ 05:26:02 DEBUG }, 05:26:02 DEBUG { 05:26:02 DEBUG "value": [ 05:26:02 DEBUG "metric": { 05:26:02 DEBUG 05:26:02 DEBUG "pod": "ds-idrepo-0" 05:26:02 DEBUG 05:26:02 DEBUG }, 05:26:02 DEBUG 05:26:02 DEBUG "value": [ 05:26:02 DEBUG [...] 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG [...] 05:26:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:26:02 DEBUG 05:26:02 DEBUG }, 05:26:02 DEBUG 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 05:26:02 DEBUG 1735363562, 05:26:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:26:02 DEBUG "14879068160" 05:26:02 DEBUG }, 05:26:02 DEBUG ] 05:26:02 DEBUG "value": [ 05:26:02 DEBUG } 05:26:02 DEBUG 1735363562, 05:26:02 DEBUG ] 05:26:02 DEBUG "0.001142682729456443" 05:26:02 DEBUG } 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363562" 05:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363562" 05:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363562" 05:26:02 INFO [http_cmd]: http status code OK 05:26:02 DEBUG --- status code --- 05:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:02 DEBUG --- http response --- 05:26:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:26:02 DEBUG ----- output ----- 05:26:02 DEBUG { 05:26:02 DEBUG "status": "success", 05:26:02 DEBUG "data": { 05:26:02 DEBUG "resultType": "vector", 05:26:02 DEBUG "result": [ 05:26:02 DEBUG { 05:26:02 DEBUG "metric": { 05:26:02 DEBUG "instance": "10.142.0.11:9100" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG [...] 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG "instance": "10.142.0.51:9100" 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 1735363562, 05:26:02 DEBUG "278169.8333333334" 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 INFO [http_cmd]: http status code OK 05:26:02 INFO [http_cmd]: http status code OK 05:26:02 DEBUG --- status code --- 05:26:02 DEBUG --- status code --- 05:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:02 DEBUG --- http response --- 05:26:02 DEBUG --- http response --- 05:26:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:26:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:26:02 DEBUG ----- output ----- 05:26:02 DEBUG ----- output ----- 05:26:02 DEBUG { 05:26:02 DEBUG { 05:26:02 DEBUG "status": "success", 05:26:02 DEBUG "status": "success", 05:26:02 DEBUG "data": { 05:26:02 DEBUG "data": { 05:26:02 DEBUG "resultType": "vector", 05:26:02 DEBUG "resultType": "vector", 05:26:02 DEBUG "result": [ 05:26:02 DEBUG "result": [ 05:26:02 DEBUG { 05:26:02 DEBUG { 05:26:02 DEBUG "metric": { 05:26:02 DEBUG "metric": { 05:26:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:26:02 DEBUG "instance": "10.142.0.11:9100" 05:26:02 DEBUG }, 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG [...] 05:26:02 DEBUG [...] 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG 05:26:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:26:02 DEBUG "instance": "10.142.0.51:9100" 05:26:02 DEBUG }, 05:26:02 DEBUG }, 05:26:02 DEBUG "value": [ 05:26:02 DEBUG "value": [ 05:26:02 DEBUG 1735363562, 05:26:02 DEBUG 1735363562, 05:26:02 DEBUG "0.11499417113914191" 05:26:02 DEBUG "19660.800000000003" 05:26:02 DEBUG ] 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 DEBUG ] 05:26:02 DEBUG ] 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:02 DEBUG } 05:26:32 INFO 05:26:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:26:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:26:32 INFO [loop_until]: OK (rc = 0) 05:26:32 DEBUG --- stdout --- 05:26:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 60m 13375Mi rcs-687d95c5b-qs2xv 1m 426Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:26:32 DEBUG --- stderr --- 05:26:32 DEBUG 05:26:34 INFO 05:26:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:26:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:26:34 INFO [loop_until]: OK (rc = 0) 05:26:34 DEBUG --- stdout --- 05:26:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 171m 1% 3423Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 95m 0% 18873Mi 66% 05:26:34 DEBUG --- stderr --- 05:26:34 DEBUG 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG --- status code --- 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 ERROR [http_cmd]: ERROR 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG --- http response --- 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 ERROR --- status code --- 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 ERROR --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363621" 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:27:01 DEBUG ----- output ----- 05:27:01 DEBUG { 05:27:01 DEBUG "status": "success", 05:27:01 DEBUG "data": { 05:27:01 DEBUG "resultType": "vector", 05:27:01 DEBUG "result": [ 05:27:01 DEBUG { 05:27:01 DEBUG "metric": { 05:27:01 DEBUG "pod": "ds-idrepo-0" 05:27:01 DEBUG }, 05:27:01 DEBUG "value": [ 05:27:01 DEBUG 05:27:01 DEBUG 05:27:01 DEBUG 05:27:01 DEBUG [...] 05:27:01 DEBUG 05:27:01 DEBUG 05:27:01 DEBUG 05:27:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:27:01 DEBUG }, 05:27:01 DEBUG "value": [ 05:27:01 DEBUG 1735363621, 05:27:01 DEBUG "831123456" 05:27:01 DEBUG ] 05:27:01 DEBUG } 05:27:01 DEBUG ] 05:27:01 DEBUG } 05:27:01 DEBUG } 05:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363621" 05:27:01 INFO [http_cmd]: http status code OK 05:27:01 DEBUG --- status code --- 05:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:01 DEBUG --- http response --- 05:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:27:01 DEBUG ----- output ----- 05:27:01 DEBUG { 05:27:01 DEBUG "status": "success", 05:27:01 DEBUG "data": { 05:27:01 DEBUG "resultType": "vector", 05:27:01 DEBUG "result": [ 05:27:01 DEBUG { 05:27:01 DEBUG "metric": { 05:27:01 DEBUG "instance": "10.142.0.11:9100" 05:27:01 DEBUG }, 05:27:01 DEBUG "value": [ 05:27:01 DEBUG 05:27:01 DEBUG 05:27:01 DEBUG 05:27:01 DEBUG [...] 05:27:01 DEBUG 05:27:01 DEBUG 05:27:01 DEBUG 05:27:01 DEBUG "instance": "10.142.0.51:9100" 05:27:01 DEBUG }, 05:27:01 DEBUG "value": [ 05:27:01 DEBUG 1735363621, 05:27:01 DEBUG "0.0011333333333330606" 05:27:01 DEBUG ] 05:27:01 DEBUG } 05:27:01 DEBUG ] 05:27:01 DEBUG } 05:27:01 DEBUG } 05:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363622" 05:27:02 INFO [http_cmd]: http status code OK 05:27:02 DEBUG --- status code --- 05:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:02 DEBUG --- http response --- 05:27:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:27:02 DEBUG ----- output ----- 05:27:02 DEBUG { 05:27:02 DEBUG "status": "success", 05:27:02 DEBUG "data": { 05:27:02 DEBUG "resultType": "vector", 05:27:02 DEBUG "result": [ 05:27:02 DEBUG { 05:27:02 DEBUG "metric": { 05:27:02 DEBUG "instance": "10.142.0.11:9100" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG [...] 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG "instance": "10.142.0.51:9100" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 1735363622, 05:27:02 DEBUG "0.0010000000000000382" 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363622" 05:27:02 INFO [http_cmd]: http status code OK 05:27:02 DEBUG --- status code --- 05:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:02 DEBUG --- http response --- 05:27:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:27:02 DEBUG ----- output ----- 05:27:02 DEBUG { 05:27:02 DEBUG "status": "success", 05:27:02 DEBUG "data": { 05:27:02 DEBUG "resultType": "vector", 05:27:02 DEBUG "result": [ 05:27:02 DEBUG { 05:27:02 DEBUG "metric": { 05:27:02 DEBUG "instance": "10.142.0.11:9100" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG [...] 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG "instance": "10.142.0.51:9100" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 1735363622, 05:27:02 DEBUG "1136949.7333333334" 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363622" 05:27:02 INFO [http_cmd]: http status code OK 05:27:02 DEBUG --- status code --- 05:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:02 DEBUG --- http response --- 05:27:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:27:02 DEBUG ----- output ----- 05:27:02 DEBUG { 05:27:02 DEBUG "status": "success", 05:27:02 DEBUG "data": { 05:27:02 DEBUG "resultType": "vector", 05:27:02 DEBUG "result": [ 05:27:02 DEBUG { 05:27:02 DEBUG "metric": { 05:27:02 DEBUG "pod": "ds-idrepo-0" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG [...] 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 1735363622, 05:27:02 DEBUG "0" 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363622" 05:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363622" 05:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363622" 05:27:02 INFO [http_cmd]: http status code OK 05:27:02 DEBUG --- status code --- 05:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:02 DEBUG --- http response --- 05:27:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:27:02 DEBUG ----- output ----- 05:27:02 DEBUG { 05:27:02 DEBUG "status": "success", 05:27:02 DEBUG "data": { 05:27:02 DEBUG "resultType": "vector", 05:27:02 DEBUG "result": [ 05:27:02 DEBUG { 05:27:02 DEBUG "metric": { 05:27:02 DEBUG "pod": "ds-idrepo-0" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG [...] 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 1735363622, 05:27:02 DEBUG "405.1434223541048" 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 INFO [http_cmd]: http status code OK 05:27:02 DEBUG --- status code --- 05:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:02 DEBUG --- http response --- 05:27:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:27:02 DEBUG ----- output ----- 05:27:02 DEBUG { 05:27:02 DEBUG "status": "success", 05:27:02 DEBUG "data": { 05:27:02 DEBUG "resultType": "vector", 05:27:02 DEBUG "result": [ 05:27:02 DEBUG { 05:27:02 DEBUG "metric": { 05:27:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG [...] 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 1735363622, 05:27:02 DEBUG "14879080448" 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 INFO [http_cmd]: http status code OK 05:27:02 DEBUG --- status code --- 05:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:02 DEBUG --- http response --- 05:27:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:27:02 DEBUG ----- output ----- 05:27:02 DEBUG { 05:27:02 DEBUG "status": "success", 05:27:02 DEBUG "data": { 05:27:02 DEBUG "resultType": "vector", 05:27:02 DEBUG "result": [ 05:27:02 DEBUG { 05:27:02 DEBUG "metric": { 05:27:02 DEBUG "pod": "ds-idrepo-0" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG [...] 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 1735363622, 05:27:02 DEBUG "0.0008483927464591693" 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363622" 05:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363622" 05:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363622" 05:27:02 INFO [http_cmd]: http status code OK 05:27:02 DEBUG --- status code --- 05:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:02 DEBUG --- http response --- 05:27:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:27:02 DEBUG ----- output ----- 05:27:02 DEBUG { 05:27:02 DEBUG "status": "success", 05:27:02 DEBUG "data": { 05:27:02 DEBUG "resultType": "vector", 05:27:02 DEBUG "result": [ 05:27:02 DEBUG { 05:27:02 DEBUG "metric": { 05:27:02 DEBUG "instance": "10.142.0.11:9100" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG [...] 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG "instance": "10.142.0.51:9100" 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 1735363622, 05:27:02 DEBUG "1155817.8666666667" 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 INFO [http_cmd]: http status code OK 05:27:02 INFO [http_cmd]: http status code OK 05:27:02 DEBUG --- status code --- 05:27:02 DEBUG --- status code --- 05:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:02 DEBUG --- http response --- 05:27:02 DEBUG --- http response --- 05:27:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:27:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:27:02 DEBUG ----- output ----- 05:27:02 DEBUG ----- output ----- 05:27:02 DEBUG { 05:27:02 DEBUG { 05:27:02 DEBUG "status": "success", 05:27:02 DEBUG "status": "success", 05:27:02 DEBUG "data": { 05:27:02 DEBUG "data": { 05:27:02 DEBUG "resultType": "vector", 05:27:02 DEBUG "resultType": "vector", 05:27:02 DEBUG "result": [ 05:27:02 DEBUG "result": [ 05:27:02 DEBUG { 05:27:02 DEBUG { 05:27:02 DEBUG "metric": { 05:27:02 DEBUG "metric": { 05:27:02 DEBUG "instance": "10.142.0.11:9100" 05:27:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:27:02 DEBUG }, 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG [...] 05:27:02 DEBUG [...] 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG 05:27:02 DEBUG "instance": "10.142.0.51:9100" 05:27:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:27:02 DEBUG }, 05:27:02 DEBUG }, 05:27:02 DEBUG "value": [ 05:27:02 DEBUG "value": [ 05:27:02 DEBUG 1735363622, 05:27:02 DEBUG 1735363622, 05:27:02 DEBUG "25258.66666666667" 05:27:02 DEBUG "0.08824861094918915" 05:27:02 DEBUG ] 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 DEBUG ] 05:27:02 DEBUG ] 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:02 DEBUG } 05:27:32 INFO 05:27:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:27:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:27:32 INFO [loop_until]: OK (rc = 0) 05:27:32 DEBUG --- stdout --- 05:27:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13376Mi rcs-687d95c5b-qs2xv 107m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:27:32 DEBUG --- stderr --- 05:27:32 DEBUG 05:27:34 INFO 05:27:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:27:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:27:34 INFO [loop_until]: OK (rc = 0) 05:27:34 DEBUG --- stdout --- 05:27:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 229m 1% 3438Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 141m 0% 18875Mi 66% 05:27:34 DEBUG --- stderr --- 05:27:34 DEBUG 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG --- status code --- 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 ERROR [http_cmd]: ERROR 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 ERROR --- status code --- 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 ERROR --- http response --- 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363681" 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:28:01 DEBUG ----- output ----- 05:28:01 DEBUG { 05:28:01 DEBUG "status": "success", 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG "data": { 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG "resultType": "vector", 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG "result": [ 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG { 05:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:01 DEBUG "metric": { 05:28:01 DEBUG "pod": "ds-idrepo-0" 05:28:01 DEBUG }, 05:28:01 DEBUG "value": [ 05:28:01 DEBUG 05:28:01 DEBUG 05:28:01 DEBUG 05:28:01 DEBUG [...] 05:28:01 DEBUG 05:28:01 DEBUG 05:28:01 DEBUG 05:28:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:28:01 DEBUG }, 05:28:01 DEBUG "value": [ 05:28:01 DEBUG 1735363681, 05:28:01 DEBUG "831123456" 05:28:01 DEBUG ] 05:28:01 DEBUG } 05:28:01 DEBUG ] 05:28:01 DEBUG } 05:28:01 DEBUG } 05:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363681" 05:28:01 INFO [http_cmd]: http status code OK 05:28:01 DEBUG --- status code --- 05:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:01 DEBUG --- http response --- 05:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:28:01 DEBUG ----- output ----- 05:28:01 DEBUG { 05:28:01 DEBUG "status": "success", 05:28:01 DEBUG "data": { 05:28:01 DEBUG "resultType": "vector", 05:28:01 DEBUG "result": [ 05:28:01 DEBUG { 05:28:01 DEBUG "metric": { 05:28:01 DEBUG "instance": "10.142.0.11:9100" 05:28:01 DEBUG }, 05:28:01 DEBUG "value": [ 05:28:01 DEBUG 05:28:01 DEBUG 05:28:01 DEBUG 05:28:01 DEBUG [...] 05:28:01 DEBUG 05:28:01 DEBUG 05:28:01 DEBUG 05:28:01 DEBUG "instance": "10.142.0.51:9100" 05:28:01 DEBUG }, 05:28:01 DEBUG "value": [ 05:28:01 DEBUG 1735363681, 05:28:01 DEBUG "0.0003999999999981205" 05:28:01 DEBUG ] 05:28:01 DEBUG } 05:28:01 DEBUG ] 05:28:01 DEBUG } 05:28:01 DEBUG } 05:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363682" 05:28:02 INFO [http_cmd]: http status code OK 05:28:02 DEBUG --- status code --- 05:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:02 DEBUG --- http response --- 05:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:28:02 DEBUG ----- output ----- 05:28:02 DEBUG { 05:28:02 DEBUG "status": "success", 05:28:02 DEBUG "data": { 05:28:02 DEBUG "resultType": "vector", 05:28:02 DEBUG "result": [ 05:28:02 DEBUG { 05:28:02 DEBUG "metric": { 05:28:02 DEBUG "instance": "10.142.0.11:9100" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG [...] 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG "instance": "10.142.0.51:9100" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 1735363682, 05:28:02 DEBUG "0.00033333333333303023" 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG } 05:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363682" 05:28:02 INFO [http_cmd]: http status code OK 05:28:02 DEBUG --- status code --- 05:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:02 DEBUG --- http response --- 05:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:28:02 DEBUG ----- output ----- 05:28:02 DEBUG { 05:28:02 DEBUG "status": "success", 05:28:02 DEBUG "data": { 05:28:02 DEBUG "resultType": "vector", 05:28:02 DEBUG "result": [ 05:28:02 DEBUG { 05:28:02 DEBUG "metric": { 05:28:02 DEBUG "instance": "10.142.0.11:9100" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG [...] 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG "instance": "10.142.0.51:9100" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 1735363682, 05:28:02 DEBUG "1979842.9333333336" 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG } 05:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363682" 05:28:02 INFO [http_cmd]: http status code OK 05:28:02 DEBUG --- status code --- 05:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:02 DEBUG --- http response --- 05:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:28:02 DEBUG ----- output ----- 05:28:02 DEBUG { 05:28:02 DEBUG "status": "success", 05:28:02 DEBUG "data": { 05:28:02 DEBUG "resultType": "vector", 05:28:02 DEBUG "result": [ 05:28:02 DEBUG { 05:28:02 DEBUG "metric": { 05:28:02 DEBUG "pod": "ds-idrepo-0" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG [...] 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 1735363682, 05:28:02 DEBUG "0" 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG } 05:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363682" 05:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363682" 05:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363682" 05:28:02 INFO [http_cmd]: http status code OK 05:28:02 DEBUG --- status code --- 05:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:02 DEBUG --- http response --- 05:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:28:02 DEBUG ----- output ----- 05:28:02 DEBUG { 05:28:02 DEBUG "status": "success", 05:28:02 DEBUG "data": { 05:28:02 DEBUG "resultType": "vector", 05:28:02 DEBUG "result": [ 05:28:02 DEBUG { 05:28:02 DEBUG "metric": { 05:28:02 DEBUG "pod": "ds-idrepo-0" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG [...] 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 1735363682, 05:28:02 DEBUG "421.6594605723698" 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG } 05:28:02 INFO [http_cmd]: http status code OK 05:28:02 DEBUG --- status code --- 05:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:02 DEBUG --- http response --- 05:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:28:02 DEBUG ----- output ----- 05:28:02 DEBUG { 05:28:02 DEBUG "status": "success", 05:28:02 DEBUG "data": { 05:28:02 DEBUG "resultType": "vector", 05:28:02 DEBUG "result": [ 05:28:02 DEBUG { 05:28:02 DEBUG "metric": { 05:28:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG [...] 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 1735363682, 05:28:02 DEBUG "14881001472" 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG } 05:28:02 INFO [http_cmd]: http status code OK 05:28:02 DEBUG --- status code --- 05:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:02 DEBUG --- http response --- 05:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:28:02 DEBUG ----- output ----- 05:28:02 DEBUG { 05:28:02 DEBUG "status": "success", 05:28:02 DEBUG "data": { 05:28:02 DEBUG "resultType": "vector", 05:28:02 DEBUG "result": [ 05:28:02 DEBUG { 05:28:02 DEBUG "metric": { 05:28:02 DEBUG "pod": "ds-idrepo-0" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG [...] 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 1735363682, 05:28:02 DEBUG "0.0008349999588270358" 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG } 05:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363682" 05:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363682" 05:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363682" 05:28:02 INFO [http_cmd]: http status code OK 05:28:02 DEBUG --- status code --- 05:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:02 DEBUG --- http response --- 05:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:28:02 DEBUG ----- output ----- 05:28:02 DEBUG { 05:28:02 DEBUG "status": "success", 05:28:02 DEBUG "data": { 05:28:02 DEBUG "resultType": "vector", 05:28:02 DEBUG "result": [ 05:28:02 DEBUG { 05:28:02 DEBUG "metric": { 05:28:02 DEBUG "instance": "10.142.0.11:9100" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG [...] 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG "instance": "10.142.0.51:9100" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 1735363682, 05:28:02 DEBUG "1998939.3000000007" 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG } 05:28:02 INFO [http_cmd]: http status code OK 05:28:02 DEBUG --- status code --- 05:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:02 DEBUG --- http response --- 05:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:28:02 DEBUG ----- output ----- 05:28:02 DEBUG { 05:28:02 DEBUG "status": "success", 05:28:02 DEBUG "data": { 05:28:02 DEBUG "resultType": "vector", 05:28:02 DEBUG "result": [ 05:28:02 DEBUG { 05:28:02 DEBUG "metric": { 05:28:02 DEBUG "instance": "10.142.0.11:9100" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG [...] 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG "instance": "10.142.0.51:9100" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 1735363682, 05:28:02 DEBUG "20206.933333333338" 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG } 05:28:02 INFO [http_cmd]: http status code OK 05:28:02 DEBUG --- status code --- 05:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:02 DEBUG --- http response --- 05:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:28:02 DEBUG ----- output ----- 05:28:02 DEBUG { 05:28:02 DEBUG "status": "success", 05:28:02 DEBUG "data": { 05:28:02 DEBUG "resultType": "vector", 05:28:02 DEBUG "result": [ 05:28:02 DEBUG { 05:28:02 DEBUG "metric": { 05:28:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG [...] 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG 05:28:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:28:02 DEBUG }, 05:28:02 DEBUG "value": [ 05:28:02 DEBUG 1735363682, 05:28:02 DEBUG "0.14674306744858603" 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG ] 05:28:02 DEBUG } 05:28:02 DEBUG } 05:28:32 INFO 05:28:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:28:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:28:32 INFO [loop_until]: OK (rc = 0) 05:28:32 DEBUG --- stdout --- 05:28:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 79m 13376Mi rcs-687d95c5b-qs2xv 81m 427Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:28:32 DEBUG --- stderr --- 05:28:32 DEBUG 05:28:34 INFO 05:28:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:28:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:28:34 INFO [loop_until]: OK (rc = 0) 05:28:34 DEBUG --- stdout --- 05:28:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 234m 1% 3446Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 183m 0% 18871Mi 66% 05:28:34 DEBUG --- stderr --- 05:28:34 DEBUG 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG --- status code --- 05:29:01 ERROR [http_cmd]: ERROR 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 ERROR --- status code --- 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- status code --- 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG --- http response --- 05:29:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 ERROR --- http response --- 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363741" 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:29:01 DEBUG ----- output ----- 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG { 05:29:01 DEBUG --- status code --- 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG "status": "success", 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG "data": { 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG "resultType": "vector", 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG "result": [ 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG { 05:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:01 DEBUG "metric": { 05:29:01 DEBUG "pod": "ds-idrepo-0" 05:29:01 DEBUG }, 05:29:01 DEBUG "value": [ 05:29:01 DEBUG 05:29:01 DEBUG 05:29:01 DEBUG 05:29:01 DEBUG [...] 05:29:01 DEBUG 05:29:01 DEBUG 05:29:01 DEBUG 05:29:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:29:01 DEBUG }, 05:29:01 DEBUG "value": [ 05:29:01 DEBUG 1735363741, 05:29:01 DEBUG "831123456" 05:29:01 DEBUG ] 05:29:01 DEBUG } 05:29:01 DEBUG ] 05:29:01 DEBUG } 05:29:01 DEBUG } 05:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363741" 05:29:01 INFO [http_cmd]: http status code OK 05:29:01 DEBUG --- status code --- 05:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:01 DEBUG --- http response --- 05:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:29:01 DEBUG ----- output ----- 05:29:01 DEBUG { 05:29:01 DEBUG "status": "success", 05:29:01 DEBUG "data": { 05:29:01 DEBUG "resultType": "vector", 05:29:01 DEBUG "result": [ 05:29:01 DEBUG { 05:29:01 DEBUG "metric": { 05:29:01 DEBUG "instance": "10.142.0.11:9100" 05:29:01 DEBUG }, 05:29:01 DEBUG "value": [ 05:29:01 DEBUG 05:29:01 DEBUG 05:29:01 DEBUG 05:29:01 DEBUG [...] 05:29:01 DEBUG 05:29:01 DEBUG 05:29:01 DEBUG 05:29:01 DEBUG "instance": "10.142.0.51:9100" 05:29:01 DEBUG }, 05:29:01 DEBUG "value": [ 05:29:01 DEBUG 1735363741, 05:29:01 DEBUG "0.0005999999999933912" 05:29:01 DEBUG ] 05:29:01 DEBUG } 05:29:01 DEBUG ] 05:29:01 DEBUG } 05:29:01 DEBUG } 05:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363742" 05:29:02 INFO [http_cmd]: http status code OK 05:29:02 DEBUG --- status code --- 05:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:02 DEBUG --- http response --- 05:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:29:02 DEBUG ----- output ----- 05:29:02 DEBUG { 05:29:02 DEBUG "status": "success", 05:29:02 DEBUG "data": { 05:29:02 DEBUG "resultType": "vector", 05:29:02 DEBUG "result": [ 05:29:02 DEBUG { 05:29:02 DEBUG "metric": { 05:29:02 DEBUG "instance": "10.142.0.11:9100" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG [...] 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG "instance": "10.142.0.51:9100" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 1735363742, 05:29:02 DEBUG "0.0009999999999995644" 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363742" 05:29:02 INFO [http_cmd]: http status code OK 05:29:02 DEBUG --- status code --- 05:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:02 DEBUG --- http response --- 05:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:29:02 DEBUG ----- output ----- 05:29:02 DEBUG { 05:29:02 DEBUG "status": "success", 05:29:02 DEBUG "data": { 05:29:02 DEBUG "resultType": "vector", 05:29:02 DEBUG "result": [ 05:29:02 DEBUG { 05:29:02 DEBUG "metric": { 05:29:02 DEBUG "instance": "10.142.0.11:9100" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG [...] 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG "instance": "10.142.0.51:9100" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 1735363742, 05:29:02 DEBUG "2293372.5" 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363742" 05:29:02 INFO [http_cmd]: http status code OK 05:29:02 DEBUG --- status code --- 05:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:02 DEBUG --- http response --- 05:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:29:02 DEBUG ----- output ----- 05:29:02 DEBUG { 05:29:02 DEBUG "status": "success", 05:29:02 DEBUG "data": { 05:29:02 DEBUG "resultType": "vector", 05:29:02 DEBUG "result": [ 05:29:02 DEBUG { 05:29:02 DEBUG "metric": { 05:29:02 DEBUG "pod": "ds-idrepo-0" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG [...] 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 1735363742, 05:29:02 DEBUG "0" 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363742" 05:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363742" 05:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363742" 05:29:02 INFO [http_cmd]: http status code OK 05:29:02 DEBUG --- status code --- 05:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:02 DEBUG --- http response --- 05:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:29:02 DEBUG ----- output ----- 05:29:02 DEBUG { 05:29:02 DEBUG "status": "success", 05:29:02 DEBUG "data": { 05:29:02 DEBUG "resultType": "vector", 05:29:02 DEBUG "result": [ 05:29:02 DEBUG { 05:29:02 DEBUG "metric": { 05:29:02 DEBUG "pod": "ds-idrepo-0" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG [...] 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 1735363742, 05:29:02 DEBUG "771.963127566602" 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:02 INFO [http_cmd]: http status code OK 05:29:02 DEBUG --- status code --- 05:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:02 DEBUG --- http response --- 05:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:29:02 INFO [http_cmd]: http status code OK 05:29:02 DEBUG ----- output ----- 05:29:02 DEBUG --- status code --- 05:29:02 DEBUG { 05:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:02 DEBUG "status": "success", 05:29:02 DEBUG --- http response --- 05:29:02 DEBUG "data": { 05:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:29:02 DEBUG "resultType": "vector", 05:29:02 DEBUG ----- output ----- 05:29:02 DEBUG "result": [ 05:29:02 DEBUG { 05:29:02 DEBUG { 05:29:02 DEBUG "status": "success", 05:29:02 DEBUG "metric": { 05:29:02 DEBUG "data": { 05:29:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:29:02 DEBUG "resultType": "vector", 05:29:02 DEBUG }, 05:29:02 DEBUG "result": [ 05:29:02 DEBUG "value": [ 05:29:02 DEBUG { 05:29:02 DEBUG 05:29:02 DEBUG "metric": { 05:29:02 DEBUG 05:29:02 DEBUG "pod": "ds-idrepo-0" 05:29:02 DEBUG 05:29:02 DEBUG }, 05:29:02 DEBUG [...] 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:29:02 DEBUG [...] 05:29:02 DEBUG }, 05:29:02 DEBUG 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 05:29:02 DEBUG 1735363742, 05:29:02 DEBUG 05:29:02 DEBUG "14880649216" 05:29:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:29:02 DEBUG ] 05:29:02 DEBUG }, 05:29:02 DEBUG } 05:29:02 DEBUG "value": [ 05:29:02 DEBUG ] 05:29:02 DEBUG 1735363742, 05:29:02 DEBUG } 05:29:02 DEBUG "0.0012511615729929664" 05:29:02 DEBUG } 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363742" 05:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363742" 05:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363742" 05:29:02 INFO [http_cmd]: http status code OK 05:29:02 DEBUG --- status code --- 05:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:02 DEBUG --- http response --- 05:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:29:02 DEBUG ----- output ----- 05:29:02 DEBUG { 05:29:02 DEBUG "status": "success", 05:29:02 DEBUG "data": { 05:29:02 DEBUG "resultType": "vector", 05:29:02 DEBUG "result": [ 05:29:02 DEBUG { 05:29:02 DEBUG "metric": { 05:29:02 DEBUG "instance": "10.142.0.11:9100" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG [...] 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG "instance": "10.142.0.51:9100" 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 1735363742, 05:29:02 DEBUG "2312509.633333334" 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:02 INFO [http_cmd]: http status code OK 05:29:02 INFO [http_cmd]: http status code OK 05:29:02 DEBUG --- status code --- 05:29:02 DEBUG --- status code --- 05:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:02 DEBUG --- http response --- 05:29:02 DEBUG --- http response --- 05:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:29:02 DEBUG ----- output ----- 05:29:02 DEBUG ----- output ----- 05:29:02 DEBUG { 05:29:02 DEBUG { 05:29:02 DEBUG "status": "success", 05:29:02 DEBUG "status": "success", 05:29:02 DEBUG "data": { 05:29:02 DEBUG "data": { 05:29:02 DEBUG "resultType": "vector", 05:29:02 DEBUG "resultType": "vector", 05:29:02 DEBUG "result": [ 05:29:02 DEBUG "result": [ 05:29:02 DEBUG { 05:29:02 DEBUG { 05:29:02 DEBUG "metric": { 05:29:02 DEBUG "metric": { 05:29:02 DEBUG "instance": "10.142.0.11:9100" 05:29:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:29:02 DEBUG }, 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG [...] 05:29:02 DEBUG [...] 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG 05:29:02 DEBUG "instance": "10.142.0.51:9100" 05:29:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:29:02 DEBUG }, 05:29:02 DEBUG }, 05:29:02 DEBUG "value": [ 05:29:02 DEBUG "value": [ 05:29:02 DEBUG 1735363742, 05:29:02 DEBUG 1735363742, 05:29:02 DEBUG "31812.26666666667" 05:29:02 DEBUG "0.17426904834256024" 05:29:02 DEBUG ] 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:02 DEBUG ] 05:29:02 DEBUG ] 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:02 DEBUG } 05:29:32 INFO 05:29:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:29:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:29:32 INFO [loop_until]: OK (rc = 0) 05:29:32 DEBUG --- stdout --- 05:29:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 65m 13375Mi rcs-687d95c5b-qs2xv 117m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:29:32 DEBUG --- stderr --- 05:29:32 DEBUG 05:29:34 INFO 05:29:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:29:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:29:34 INFO [loop_until]: OK (rc = 0) 05:29:34 DEBUG --- stdout --- 05:29:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 228m 1% 3461Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 207m 0% 18870Mi 66% 05:29:34 DEBUG --- stderr --- 05:29:34 DEBUG 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 ERROR [http_cmd]: ERROR 05:30:01 ERROR --- status code --- 05:30:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:30:01 ERROR --- http response --- 05:30:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:30:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG --- status code --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363801" 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG --- status code --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG --- http response --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG --- http response --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG --- status code --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- status code --- 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:30:01 DEBUG ----- output ----- 05:30:01 DEBUG { 05:30:01 DEBUG "status": "success", 05:30:01 DEBUG "data": { 05:30:01 DEBUG "resultType": "vector", 05:30:01 DEBUG "result": [ 05:30:01 DEBUG { 05:30:01 DEBUG "metric": { 05:30:01 DEBUG "pod": "ds-idrepo-0" 05:30:01 DEBUG }, 05:30:01 DEBUG "value": [ 05:30:01 DEBUG 05:30:01 DEBUG 05:30:01 DEBUG 05:30:01 DEBUG [...] 05:30:01 DEBUG 05:30:01 DEBUG 05:30:01 DEBUG 05:30:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:30:01 DEBUG }, 05:30:01 DEBUG "value": [ 05:30:01 DEBUG 1735363801, 05:30:01 DEBUG "831123456" 05:30:01 DEBUG ] 05:30:01 DEBUG } 05:30:01 DEBUG ] 05:30:01 DEBUG } 05:30:01 DEBUG } 05:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363801" 05:30:01 INFO [http_cmd]: http status code OK 05:30:01 DEBUG --- status code --- 05:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:01 DEBUG --- http response --- 05:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:30:01 DEBUG ----- output ----- 05:30:01 DEBUG { 05:30:01 DEBUG "status": "success", 05:30:01 DEBUG "data": { 05:30:01 DEBUG "resultType": "vector", 05:30:01 DEBUG "result": [ 05:30:01 DEBUG { 05:30:01 DEBUG "metric": { 05:30:01 DEBUG "instance": "10.142.0.11:9100" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG [...] 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG "instance": "10.142.0.51:9100" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 1735363801, 05:30:02 DEBUG "0.0003999999999981205" 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363802" 05:30:02 INFO [http_cmd]: http status code OK 05:30:02 DEBUG --- status code --- 05:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:02 DEBUG --- http response --- 05:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:30:02 DEBUG ----- output ----- 05:30:02 DEBUG { 05:30:02 DEBUG "status": "success", 05:30:02 DEBUG "data": { 05:30:02 DEBUG "resultType": "vector", 05:30:02 DEBUG "result": [ 05:30:02 DEBUG { 05:30:02 DEBUG "metric": { 05:30:02 DEBUG "instance": "10.142.0.11:9100" 05:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363802" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG [...] 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG "instance": "10.142.0.51:9100" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 1735363802, 05:30:02 DEBUG "0.0003333333333335039" 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 INFO [http_cmd]: http status code OK 05:30:02 DEBUG --- status code --- 05:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:02 DEBUG --- http response --- 05:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:30:02 DEBUG ----- output ----- 05:30:02 DEBUG { 05:30:02 DEBUG "status": "success", 05:30:02 DEBUG "data": { 05:30:02 DEBUG "resultType": "vector", 05:30:02 DEBUG "result": [ 05:30:02 DEBUG { 05:30:02 DEBUG "metric": { 05:30:02 DEBUG "instance": "10.142.0.11:9100" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG [...] 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG "instance": "10.142.0.51:9100" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 1735363802, 05:30:02 DEBUG "2380978.1666666674" 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363802" 05:30:02 INFO [http_cmd]: http status code OK 05:30:02 DEBUG --- status code --- 05:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:02 DEBUG --- http response --- 05:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:30:02 DEBUG ----- output ----- 05:30:02 DEBUG { 05:30:02 DEBUG "status": "success", 05:30:02 DEBUG "data": { 05:30:02 DEBUG "resultType": "vector", 05:30:02 DEBUG "result": [ 05:30:02 DEBUG { 05:30:02 DEBUG "metric": { 05:30:02 DEBUG "pod": "ds-idrepo-0" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG [...] 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 1735363802, 05:30:02 DEBUG "0" 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363802" 05:30:02 INFO [http_cmd]: http status code OK 05:30:02 DEBUG --- status code --- 05:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:02 DEBUG --- http response --- 05:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:30:02 DEBUG ----- output ----- 05:30:02 DEBUG { 05:30:02 DEBUG "status": "success", 05:30:02 DEBUG "data": { 05:30:02 DEBUG "resultType": "vector", 05:30:02 DEBUG "result": [ 05:30:02 DEBUG { 05:30:02 DEBUG "metric": { 05:30:02 DEBUG "pod": "ds-idrepo-0" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG [...] 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363802" 05:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363802" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 1735363802, 05:30:02 DEBUG "905.1933810837188" 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 INFO [http_cmd]: http status code OK 05:30:02 DEBUG --- status code --- 05:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:02 DEBUG --- http response --- 05:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:30:02 DEBUG ----- output ----- 05:30:02 DEBUG { 05:30:02 DEBUG "status": "success", 05:30:02 DEBUG "data": { 05:30:02 DEBUG "resultType": "vector", 05:30:02 INFO [http_cmd]: http status code OK 05:30:02 DEBUG "result": [ 05:30:02 DEBUG --- status code --- 05:30:02 DEBUG { 05:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:02 DEBUG "metric": { 05:30:02 DEBUG --- http response --- 05:30:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG [...] 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 1735363802, 05:30:02 DEBUG "14880956416" 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363802" 05:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:30:02 DEBUG ----- output ----- 05:30:02 DEBUG { 05:30:02 DEBUG "status": "success", 05:30:02 DEBUG "data": { 05:30:02 DEBUG "resultType": "vector", 05:30:02 DEBUG "result": [ 05:30:02 DEBUG { 05:30:02 DEBUG "metric": { 05:30:02 DEBUG "pod": "ds-idrepo-0" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG [...] 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 1735363802, 05:30:02 DEBUG "0.0012811258193423897" 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363802" 05:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363802" 05:30:02 INFO [http_cmd]: http status code OK 05:30:02 DEBUG --- status code --- 05:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:02 DEBUG --- http response --- 05:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:30:02 DEBUG ----- output ----- 05:30:02 DEBUG { 05:30:02 DEBUG "status": "success", 05:30:02 DEBUG "data": { 05:30:02 DEBUG "resultType": "vector", 05:30:02 DEBUG "result": [ 05:30:02 DEBUG { 05:30:02 DEBUG "metric": { 05:30:02 DEBUG "instance": "10.142.0.11:9100" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG [...] 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG "instance": "10.142.0.51:9100" 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 1735363802, 05:30:02 DEBUG "2399850.933333334" 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 INFO [http_cmd]: http status code OK 05:30:02 INFO [http_cmd]: http status code OK 05:30:02 DEBUG --- status code --- 05:30:02 DEBUG --- status code --- 05:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:02 DEBUG --- http response --- 05:30:02 DEBUG --- http response --- 05:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:30:02 DEBUG ----- output ----- 05:30:02 DEBUG ----- output ----- 05:30:02 DEBUG { 05:30:02 DEBUG { 05:30:02 DEBUG "status": "success", 05:30:02 DEBUG "status": "success", 05:30:02 DEBUG "data": { 05:30:02 DEBUG "data": { 05:30:02 DEBUG "resultType": "vector", 05:30:02 DEBUG "resultType": "vector", 05:30:02 DEBUG "result": [ 05:30:02 DEBUG "result": [ 05:30:02 DEBUG { 05:30:02 DEBUG { 05:30:02 DEBUG "metric": { 05:30:02 DEBUG "metric": { 05:30:02 DEBUG "instance": "10.142.0.11:9100" 05:30:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:30:02 DEBUG }, 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG [...] 05:30:02 DEBUG [...] 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG 05:30:02 DEBUG "instance": "10.142.0.51:9100" 05:30:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:30:02 DEBUG }, 05:30:02 DEBUG }, 05:30:02 DEBUG "value": [ 05:30:02 DEBUG "value": [ 05:30:02 DEBUG 1735363802, 05:30:02 DEBUG 1735363802, 05:30:02 DEBUG "18568.533333333336" 05:30:02 DEBUG "0.6373649354671416" 05:30:02 DEBUG ] 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 DEBUG ] 05:30:02 DEBUG ] 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:02 DEBUG } 05:30:32 INFO 05:30:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:30:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:30:32 INFO [loop_until]: OK (rc = 0) 05:30:32 DEBUG --- stdout --- 05:30:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 193m 13376Mi rcs-687d95c5b-qs2xv 136m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:30:32 DEBUG --- stderr --- 05:30:32 DEBUG 05:30:34 INFO 05:30:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:30:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:30:34 INFO [loop_until]: OK (rc = 0) 05:30:34 DEBUG --- stdout --- 05:30:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 222m 1% 3474Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 202m 0% 18872Mi 66% 05:30:34 DEBUG --- stderr --- 05:30:34 DEBUG 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 ERROR [http_cmd]: ERROR 05:31:01 ERROR --- status code --- 05:31:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:31:01 ERROR --- http response --- 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363861" 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:01 INFO [http_cmd]: http status code OK 05:31:01 DEBUG --- status code --- 05:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:01 DEBUG --- http response --- 05:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:31:01 DEBUG ----- output ----- 05:31:01 DEBUG { 05:31:01 DEBUG "status": "success", 05:31:01 DEBUG "data": { 05:31:01 DEBUG "resultType": "vector", 05:31:01 DEBUG "result": [ 05:31:01 DEBUG { 05:31:01 DEBUG "metric": { 05:31:01 DEBUG "pod": "ds-idrepo-0" 05:31:01 DEBUG }, 05:31:01 DEBUG "value": [ 05:31:01 DEBUG 05:31:01 DEBUG 05:31:01 DEBUG 05:31:01 DEBUG [...] 05:31:01 DEBUG 05:31:01 DEBUG 05:31:01 DEBUG 05:31:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:31:01 DEBUG }, 05:31:01 DEBUG "value": [ 05:31:01 DEBUG 1735363861, 05:31:01 DEBUG "831492096" 05:31:01 DEBUG ] 05:31:01 DEBUG } 05:31:01 DEBUG ] 05:31:01 DEBUG } 05:31:01 DEBUG } 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363862" 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG --- http response --- 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG { 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG "data": { 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG "result": [ 05:31:02 DEBUG { 05:31:02 DEBUG "metric": { 05:31:02 DEBUG "instance": "10.142.0.11:9100" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG [...] 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG "instance": "10.142.0.51:9100" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG "0.00040000000000191" 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG } 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363862" 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363862" 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG --- http response --- 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG { 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG "data": { 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG "result": [ 05:31:02 DEBUG { 05:31:02 DEBUG "metric": { 05:31:02 DEBUG "instance": "10.142.0.11:9100" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG [...] 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG "instance": "10.142.0.51:9100" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG "0.00033333333333303023" 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG } 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG --- http response --- 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG { 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG "data": { 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG "result": [ 05:31:02 DEBUG { 05:31:02 DEBUG "metric": { 05:31:02 DEBUG "instance": "10.142.0.11:9100" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG [...] 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG "instance": "10.142.0.51:9100" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG "2376513.0666666673" 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG } 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363862" 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG --- http response --- 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG { 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG "data": { 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG "result": [ 05:31:02 DEBUG { 05:31:02 DEBUG "metric": { 05:31:02 DEBUG "pod": "ds-idrepo-0" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG [...] 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG "0" 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG } 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363862" 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG --- http response --- 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG { 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG "data": { 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG "result": [ 05:31:02 DEBUG { 05:31:02 DEBUG "metric": { 05:31:02 DEBUG "pod": "ds-idrepo-0" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG [...] 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG "1340.5216689514032" 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG } 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363862" 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363862" 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG --- http response --- 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG { 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG "data": { 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG "result": [ 05:31:02 DEBUG { 05:31:02 DEBUG "metric": { 05:31:02 DEBUG "pod": "ds-idrepo-0" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG [...] 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG "0.008701850889219567" 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG } 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG --- http response --- 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG { 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG "data": { 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG "result": [ 05:31:02 DEBUG { 05:31:02 DEBUG "metric": { 05:31:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG [...] 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG "14882185216" 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG } 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363862" 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG --- http response --- 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG { 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG "data": { 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG "result": [ 05:31:02 DEBUG { 05:31:02 DEBUG "metric": { 05:31:02 DEBUG "instance": "10.142.0.11:9100" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG [...] 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG "instance": "10.142.0.51:9100" 05:31:02 DEBUG }, 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG "2393799.1" 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363862" 05:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363862" 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG } 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG --- http response --- 05:31:02 INFO [http_cmd]: http status code OK 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:31:02 DEBUG --- status code --- 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:02 DEBUG { 05:31:02 DEBUG --- http response --- 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:31:02 DEBUG "data": { 05:31:02 DEBUG ----- output ----- 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG { 05:31:02 DEBUG "result": [ 05:31:02 DEBUG "status": "success", 05:31:02 DEBUG { 05:31:02 DEBUG "data": { 05:31:02 DEBUG "metric": { 05:31:02 DEBUG "resultType": "vector", 05:31:02 DEBUG "instance": "10.142.0.11:9100" 05:31:02 DEBUG "result": [ 05:31:02 DEBUG }, 05:31:02 DEBUG { 05:31:02 DEBUG "value": [ 05:31:02 DEBUG "metric": { 05:31:02 DEBUG 05:31:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:31:02 DEBUG 05:31:02 DEBUG }, 05:31:02 DEBUG 05:31:02 DEBUG "value": [ 05:31:02 DEBUG [...] 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG 05:31:02 DEBUG [...] 05:31:02 DEBUG "instance": "10.142.0.51:9100" 05:31:02 DEBUG 05:31:02 DEBUG }, 05:31:02 DEBUG 05:31:02 DEBUG "value": [ 05:31:02 DEBUG 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:31:02 DEBUG "26487.46666666667" 05:31:02 DEBUG }, 05:31:02 DEBUG ] 05:31:02 DEBUG "value": [ 05:31:02 DEBUG } 05:31:02 DEBUG 1735363862, 05:31:02 DEBUG ] 05:31:02 DEBUG "0.1289063310583877" 05:31:02 DEBUG } 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG } 05:31:02 DEBUG ] 05:31:02 DEBUG } 05:31:02 DEBUG } 05:31:32 INFO 05:31:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:31:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:31:32 INFO [loop_until]: OK (rc = 0) 05:31:32 DEBUG --- stdout --- 05:31:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 132m 13376Mi rcs-687d95c5b-qs2xv 122m 426Mi rcs-687d95c5b-xxxmk 5m 395Mi 05:31:32 DEBUG --- stderr --- 05:31:32 DEBUG 05:31:34 INFO 05:31:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:31:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:31:34 INFO [loop_until]: OK (rc = 0) 05:31:34 DEBUG --- stdout --- 05:31:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 187m 1% 3474Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 197m 0% 18870Mi 66% 05:31:34 DEBUG --- stderr --- 05:31:34 DEBUG 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- status code --- 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 ERROR [http_cmd]: ERROR 05:32:01 DEBUG --- status code --- 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 ERROR --- status code --- 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 ERROR --- http response --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363921" 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG --- http response --- 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 INFO [http_cmd]: http status code OK 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG --- status code --- 05:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:01 DEBUG --- http response --- 05:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:32:01 DEBUG ----- output ----- 05:32:01 DEBUG { 05:32:01 DEBUG "status": "success", 05:32:01 DEBUG "data": { 05:32:01 DEBUG "resultType": "vector", 05:32:01 DEBUG "result": [ 05:32:01 DEBUG { 05:32:01 DEBUG "metric": { 05:32:01 DEBUG "pod": "ds-idrepo-0" 05:32:01 DEBUG }, 05:32:01 DEBUG "value": [ 05:32:01 DEBUG 05:32:01 DEBUG 05:32:01 DEBUG 05:32:01 DEBUG [...] 05:32:01 DEBUG 05:32:01 DEBUG 05:32:01 DEBUG 05:32:01 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:32:01 DEBUG }, 05:32:01 DEBUG "value": [ 05:32:01 DEBUG 1735363921, 05:32:01 DEBUG "831582208" 05:32:01 DEBUG ] 05:32:01 DEBUG } 05:32:01 DEBUG ] 05:32:01 DEBUG } 05:32:01 DEBUG } 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363922" 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "instance": "10.142.0.11:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "instance": "10.142.0.51:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "0.0013000000000033653" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363922" 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363922" 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "instance": "10.142.0.11:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "instance": "10.142.0.51:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "0.0016666666666675196" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "instance": "10.142.0.11:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "instance": "10.142.0.51:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "1426192.4333333336" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363922" 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "pod": "ds-idrepo-0" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "0" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363922" 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "pod": "ds-idrepo-0" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "1041.1681084945765" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363922" 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363922" 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "pod": "ds-idrepo-0" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "0.0013687559206313196" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363922" 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "14878179328" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "instance": "10.142.0.11:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "instance": "10.142.0.51:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "1445759.8333333335" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363922" 05:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363922" 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "instance": "10.142.0.11:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "instance": "10.142.0.51:9100" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "20343.46666666667" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:02 INFO [http_cmd]: http status code OK 05:32:02 DEBUG --- status code --- 05:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:02 DEBUG --- http response --- 05:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:32:02 DEBUG ----- output ----- 05:32:02 DEBUG { 05:32:02 DEBUG "status": "success", 05:32:02 DEBUG "data": { 05:32:02 DEBUG "resultType": "vector", 05:32:02 DEBUG "result": [ 05:32:02 DEBUG { 05:32:02 DEBUG "metric": { 05:32:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG [...] 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG 05:32:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:32:02 DEBUG }, 05:32:02 DEBUG "value": [ 05:32:02 DEBUG 1735363922, 05:32:02 DEBUG "0.11427466719552204" 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG ] 05:32:02 DEBUG } 05:32:02 DEBUG } 05:32:32 INFO 05:32:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:32:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:32:32 INFO [loop_until]: OK (rc = 0) 05:32:32 DEBUG --- stdout --- 05:32:32 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 67m 13376Mi rcs-687d95c5b-qs2xv 36m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:32:32 DEBUG --- stderr --- 05:32:32 DEBUG 05:32:34 INFO 05:32:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:32:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:32:34 INFO [loop_until]: OK (rc = 0) 05:32:34 DEBUG --- stdout --- 05:32:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 117m 0% 3440Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 151m 0% 18870Mi 66% 05:32:34 DEBUG --- stderr --- 05:32:34 DEBUG 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG --- status code --- 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 INFO [http_cmd]: http status code OK 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- status code --- 05:33:01 ERROR [http_cmd]: ERROR 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 ERROR --- status code --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- status code --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG --- http response --- 05:33:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 ERROR --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG --- http response --- 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735363981" 05:33:01 INFO [http_cmd]: http status code OK 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG --- status code --- 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:33:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "pod": "ds-idrepo-0" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363981, 05:33:02 DEBUG "831582208" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363982" 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "instance": "10.142.0.11:9100" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "instance": "10.142.0.51:9100" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "0.0004666666666632107" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363982" 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363982" 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "instance": "10.142.0.11:9100" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "instance": "10.142.0.51:9100" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "0.0003333333333335039" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "instance": "10.142.0.11:9100" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "instance": "10.142.0.51:9100" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "972133.8000000003" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363982" 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "pod": "ds-idrepo-0" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "0" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363982" 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735363982" 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "pod": "ds-idrepo-0" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "749.81310215592" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735363982" 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "pod": "ds-idrepo-0" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "0.001291142746136818" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "14880227328" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363982" 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735363982" 05:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735363982" 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "instance": "10.142.0.11:9100" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "instance": "10.142.0.51:9100" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "991910.3333333337" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG } 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG { 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG "data": { 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "result": [ 05:33:02 DEBUG { 05:33:02 DEBUG "metric": { 05:33:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:33:02 INFO [http_cmd]: http status code OK 05:33:02 DEBUG }, 05:33:02 DEBUG --- status code --- 05:33:02 DEBUG "value": [ 05:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:02 DEBUG 05:33:02 DEBUG --- http response --- 05:33:02 DEBUG 05:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:33:02 DEBUG 05:33:02 DEBUG ----- output ----- 05:33:02 DEBUG [...] 05:33:02 DEBUG { 05:33:02 DEBUG 05:33:02 DEBUG "status": "success", 05:33:02 DEBUG 05:33:02 DEBUG "data": { 05:33:02 DEBUG 05:33:02 DEBUG "resultType": "vector", 05:33:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:33:02 DEBUG "result": [ 05:33:02 DEBUG }, 05:33:02 DEBUG { 05:33:02 DEBUG "value": [ 05:33:02 DEBUG "metric": { 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "instance": "10.142.0.11:9100" 05:33:02 DEBUG "0.031772879154206624" 05:33:02 DEBUG }, 05:33:02 DEBUG ] 05:33:02 DEBUG "value": [ 05:33:02 DEBUG } 05:33:02 DEBUG 05:33:02 DEBUG ] 05:33:02 DEBUG 05:33:02 DEBUG } 05:33:02 DEBUG 05:33:02 DEBUG } 05:33:02 DEBUG [...] 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG 05:33:02 DEBUG "instance": "10.142.0.51:9100" 05:33:02 DEBUG }, 05:33:02 DEBUG "value": [ 05:33:02 DEBUG 1735363982, 05:33:02 DEBUG "21572.26666666667" 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG ] 05:33:02 DEBUG } 05:33:02 DEBUG } 05:33:32 INFO 05:33:32 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:33:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:33 INFO [loop_until]: OK (rc = 0) 05:33:33 DEBUG --- stdout --- 05:33:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 24m 13376Mi rcs-687d95c5b-qs2xv 71m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:33:33 DEBUG --- stderr --- 05:33:33 DEBUG 05:33:34 INFO 05:33:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:33:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:35 INFO [loop_until]: OK (rc = 0) 05:33:35 DEBUG --- stdout --- 05:33:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 106m 0% 3460Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 131m 0% 18870Mi 66% 05:33:35 DEBUG --- stderr --- 05:33:35 DEBUG 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364041" 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 ERROR [http_cmd]: ERROR 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- http response --- 05:34:01 ERROR --- status code --- 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG --- status code --- 05:34:01 INFO [http_cmd]: http status code OK 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG --- status code --- 05:34:01 ERROR --- http response --- 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- status code --- 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG --- http response --- 05:34:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG --- http response --- 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG --- http response --- 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "pod": "ds-idrepo-0" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "831606784" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364042" 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "instance": "10.142.0.11:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "instance": "10.142.0.51:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "0.0004999999999976505" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364042" 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "instance": "10.142.0.11:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "instance": "10.142.0.51:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "0.0003333333333335039" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "instance": "10.142.0.11:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "instance": "10.142.0.51:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "74582.70000000004" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "pod": "ds-idrepo-0" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "0" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364042" 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364042" 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "pod": "ds-idrepo-0" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "619.421824007512" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "pod": "ds-idrepo-0" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "0.002793331144150998" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364042" 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "14878392320" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364042" 05:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364042" 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "instance": "10.142.0.11:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "instance": "10.142.0.51:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "93608.16666666672" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "0.037287568755548554" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:02 INFO [http_cmd]: http status code OK 05:34:02 DEBUG --- status code --- 05:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:02 DEBUG --- http response --- 05:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:34:02 DEBUG ----- output ----- 05:34:02 DEBUG { 05:34:02 DEBUG "status": "success", 05:34:02 DEBUG "data": { 05:34:02 DEBUG "resultType": "vector", 05:34:02 DEBUG "result": [ 05:34:02 DEBUG { 05:34:02 DEBUG "metric": { 05:34:02 DEBUG "instance": "10.142.0.11:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG [...] 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG 05:34:02 DEBUG "instance": "10.142.0.51:9100" 05:34:02 DEBUG }, 05:34:02 DEBUG "value": [ 05:34:02 DEBUG 1735364042, 05:34:02 DEBUG "27033.600000000006" 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG ] 05:34:02 DEBUG } 05:34:02 DEBUG } 05:34:33 INFO 05:34:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:34:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:34:33 INFO [loop_until]: OK (rc = 0) 05:34:33 DEBUG --- stdout --- 05:34:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 6m 13370Mi rcs-687d95c5b-qs2xv 56m 426Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:34:33 DEBUG --- stderr --- 05:34:33 DEBUG 05:34:35 INFO 05:34:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:34:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:34:35 INFO [loop_until]: OK (rc = 0) 05:34:35 DEBUG --- stdout --- 05:34:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 115m 0% 3435Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 96m 0% 18863Mi 66% 05:34:35 DEBUG --- stderr --- 05:34:35 DEBUG 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364101" 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 ERROR [http_cmd]: ERROR 05:35:01 ERROR --- status code --- 05:35:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:35:01 ERROR --- http response --- 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 INFO [http_cmd]: http status code OK 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- status code --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:01 DEBUG --- http response --- 05:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "pod": "ds-idrepo-0" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "831606784" 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG ] 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG } 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG ] 05:35:02 DEBUG --- http response --- 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG } 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG } 05:35:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364102" 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "instance": "10.142.0.11:9100" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "instance": "10.142.0.51:9100" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "0.0004999999999995454" 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG } 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364102" 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "instance": "10.142.0.11:9100" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "instance": "10.142.0.51:9100" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "0.0003333333333335039" 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG } 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "instance": "10.142.0.11:9100" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "instance": "10.142.0.51:9100" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "1388185.3000000003" 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG } 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "pod": "ds-idrepo-0" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "0" 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG } 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364102" 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364102" 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "pod": "ds-idrepo-0" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "1070.4731463505968" 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG } 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "pod": "ds-idrepo-0" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "0.0012462694855679567" 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG } 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364102" 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "14875881472" 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG } 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364102" 05:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364102" 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "instance": "10.142.0.11:9100" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "instance": "10.142.0.51:9100" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "1406964.5333333334" 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG } 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG { 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "data": { 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "result": [ 05:35:02 DEBUG { 05:35:02 DEBUG "metric": { 05:35:02 DEBUG "instance": "10.142.0.11:9100" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 INFO [http_cmd]: http status code OK 05:35:02 DEBUG 05:35:02 DEBUG --- status code --- 05:35:02 DEBUG 05:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:02 DEBUG 05:35:02 DEBUG --- http response --- 05:35:02 DEBUG [...] 05:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:35:02 DEBUG 05:35:02 DEBUG ----- output ----- 05:35:02 DEBUG 05:35:02 DEBUG { 05:35:02 DEBUG 05:35:02 DEBUG "status": "success", 05:35:02 DEBUG "instance": "10.142.0.51:9100" 05:35:02 DEBUG "data": { 05:35:02 DEBUG }, 05:35:02 DEBUG "resultType": "vector", 05:35:02 DEBUG "value": [ 05:35:02 DEBUG "result": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG { 05:35:02 DEBUG "20070.4" 05:35:02 DEBUG "metric": { 05:35:02 DEBUG ] 05:35:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:35:02 DEBUG } 05:35:02 DEBUG }, 05:35:02 DEBUG ] 05:35:02 DEBUG "value": [ 05:35:02 DEBUG } 05:35:02 DEBUG 05:35:02 DEBUG } 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG [...] 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG 05:35:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:35:02 DEBUG }, 05:35:02 DEBUG "value": [ 05:35:02 DEBUG 1735364102, 05:35:02 DEBUG "0.145770030333898" 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG ] 05:35:02 DEBUG } 05:35:02 DEBUG } 05:35:33 INFO 05:35:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:35:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:35:33 INFO [loop_until]: OK (rc = 0) 05:35:33 DEBUG --- stdout --- 05:35:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 113m 13370Mi rcs-687d95c5b-qs2xv 108m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:35:33 DEBUG --- stderr --- 05:35:33 DEBUG 05:35:35 INFO 05:35:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:35:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:35:35 INFO [loop_until]: OK (rc = 0) 05:35:35 DEBUG --- stdout --- 05:35:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 175m 1% 3439Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 169m 0% 18866Mi 66% 05:35:35 DEBUG --- stderr --- 05:35:35 DEBUG 05:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364161" 05:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364161" 05:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364161" 05:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364161" 05:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364161" 05:36:01 INFO [http_cmd]: http status code OK 05:36:01 DEBUG --- status code --- 05:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:01 INFO [http_cmd]: http status code OK 05:36:01 DEBUG --- http response --- 05:36:01 DEBUG --- status code --- 05:36:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:01 DEBUG --- http response --- 05:36:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:01 INFO [http_cmd]: http status code OK 05:36:01 DEBUG --- status code --- 05:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:01 DEBUG --- http response --- 05:36:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:01 INFO [http_cmd]: http status code OK 05:36:01 DEBUG --- status code --- 05:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:01 INFO [http_cmd]: http status code OK 05:36:01 DEBUG --- http response --- 05:36:01 DEBUG --- status code --- 05:36:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:01 DEBUG --- http response --- 05:36:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364161" 05:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364161" 05:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364161" 05:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364161" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364162" 05:36:02 ERROR [http_cmd]: ERROR 05:36:02 ERROR --- status code --- 05:36:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:36:02 ERROR --- http response --- 05:36:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:36:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG --- status code --- 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- http response --- 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG --- http response --- 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG { 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "data": { 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "result": [ 05:36:02 DEBUG { 05:36:02 DEBUG "metric": { 05:36:02 DEBUG "pod": "ds-idrepo-0" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG "831606784" 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364162" 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG { 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "data": { 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "result": [ 05:36:02 DEBUG { 05:36:02 DEBUG "metric": { 05:36:02 DEBUG "instance": "10.142.0.11:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG "instance": "10.142.0.51:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG "0.0004666666666632107" 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364162" 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG { 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "data": { 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "result": [ 05:36:02 DEBUG { 05:36:02 DEBUG "metric": { 05:36:02 DEBUG "instance": "10.142.0.11:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG "instance": "10.142.0.51:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG "79099.76666666668" 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG { 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "data": { 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "result": [ 05:36:02 DEBUG { 05:36:02 DEBUG "metric": { 05:36:02 DEBUG "instance": "10.142.0.11:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG "instance": "10.142.0.51:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG "0.0006666666666679552" 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG { 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "data": { 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "result": [ 05:36:02 DEBUG { 05:36:02 DEBUG "metric": { 05:36:02 DEBUG "pod": "ds-idrepo-0" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG "0" 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364162" 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG { 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "data": { 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "result": [ 05:36:02 DEBUG { 05:36:02 DEBUG "metric": { 05:36:02 DEBUG "pod": "ds-idrepo-0" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG "869.1396277892009" 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364162" 05:36:02 DEBUG --- status code --- 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:36:02 DEBUG { 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG { 05:36:02 DEBUG "data": { 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "data": { 05:36:02 DEBUG "result": [ 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG { 05:36:02 DEBUG "result": [ 05:36:02 DEBUG "metric": { 05:36:02 DEBUG { 05:36:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:36:02 DEBUG "metric": { 05:36:02 DEBUG }, 05:36:02 DEBUG "pod": "ds-idrepo-0" 05:36:02 DEBUG "value": [ 05:36:02 DEBUG }, 05:36:02 DEBUG 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:36:02 DEBUG 05:36:02 DEBUG }, 05:36:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:36:02 DEBUG "value": [ 05:36:02 DEBUG }, 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG "14876483584" 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG ] 05:36:02 DEBUG "0.0010708759065785013" 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364162" 05:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364162" 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG { 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "data": { 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "result": [ 05:36:02 DEBUG { 05:36:02 DEBUG "metric": { 05:36:02 DEBUG "instance": "10.142.0.11:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG "instance": "10.142.0.51:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG "96459.73333333335" 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 INFO [http_cmd]: http status code OK 05:36:02 DEBUG --- status code --- 05:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:02 DEBUG --- http response --- 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG ----- output ----- 05:36:02 DEBUG { 05:36:02 DEBUG { 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "status": "success", 05:36:02 DEBUG "data": { 05:36:02 DEBUG "data": { 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "resultType": "vector", 05:36:02 DEBUG "result": [ 05:36:02 DEBUG "result": [ 05:36:02 DEBUG { 05:36:02 DEBUG { 05:36:02 DEBUG "metric": { 05:36:02 DEBUG "metric": { 05:36:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:36:02 DEBUG "instance": "10.142.0.11:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG [...] 05:36:02 DEBUG [...] 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG 05:36:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:36:02 DEBUG "instance": "10.142.0.51:9100" 05:36:02 DEBUG }, 05:36:02 DEBUG }, 05:36:02 DEBUG "value": [ 05:36:02 DEBUG "value": [ 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG 1735364162, 05:36:02 DEBUG "0.14423883427608516" 05:36:02 DEBUG "18295.466666666667" 05:36:02 DEBUG ] 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 DEBUG ] 05:36:02 DEBUG ] 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:02 DEBUG } 05:36:33 INFO 05:36:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:36:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:36:33 INFO [loop_until]: OK (rc = 0) 05:36:33 DEBUG --- stdout --- 05:36:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13370Mi rcs-687d95c5b-qs2xv 1m 426Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:36:33 DEBUG --- stderr --- 05:36:33 DEBUG 05:36:35 INFO 05:36:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:36:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:36:35 INFO [loop_until]: OK (rc = 0) 05:36:35 DEBUG --- stdout --- 05:36:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 120m 0% 3465Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 94m 0% 18869Mi 66% 05:36:35 DEBUG --- stderr --- 05:36:35 DEBUG 05:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364221" 05:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364221" 05:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364221" 05:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364221" 05:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364221" 05:37:01 INFO [http_cmd]: http status code OK 05:37:01 DEBUG --- status code --- 05:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:01 DEBUG --- http response --- 05:37:01 INFO [http_cmd]: http status code OK 05:37:01 DEBUG --- status code --- 05:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:01 DEBUG --- http response --- 05:37:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:01 INFO [http_cmd]: http status code OK 05:37:01 INFO [http_cmd]: http status code OK 05:37:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:01 DEBUG --- status code --- 05:37:01 DEBUG --- status code --- 05:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:01 DEBUG --- http response --- 05:37:01 INFO [http_cmd]: http status code OK 05:37:01 DEBUG --- http response --- 05:37:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:01 DEBUG --- status code --- 05:37:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:01 DEBUG --- http response --- 05:37:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 ERROR [http_cmd]: ERROR 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 ERROR --- status code --- 05:37:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:37:02 ERROR --- http response --- 05:37:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:37:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364222" 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG "data": { 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG "result": [ 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 DEBUG "metric": { 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG "pod": "ds-idrepo-0" 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG }, 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG "value": [ 05:37:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG "831606784" 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG { 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG "data": { 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "result": [ 05:37:02 DEBUG { 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "instance": "10.142.0.11:9100" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG "instance": "10.142.0.51:9100" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG "0.0010333333333354254" 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364222" 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG { 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG "data": { 05:37:02 DEBUG { 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG "result": [ 05:37:02 DEBUG "data": { 05:37:02 DEBUG { 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "result": [ 05:37:02 DEBUG "instance": "10.142.0.11:9100" 05:37:02 DEBUG { 05:37:02 DEBUG }, 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "value": [ 05:37:02 DEBUG "instance": "10.142.0.11:9100" 05:37:02 DEBUG 05:37:02 DEBUG }, 05:37:02 DEBUG 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG "instance": "10.142.0.51:9100" 05:37:02 DEBUG 05:37:02 DEBUG }, 05:37:02 DEBUG 05:37:02 DEBUG "value": [ 05:37:02 DEBUG "instance": "10.142.0.51:9100" 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG }, 05:37:02 DEBUG "77151.00000000001" 05:37:02 DEBUG "value": [ 05:37:02 DEBUG ] 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG } 05:37:02 DEBUG "0.0009999999999990906" 05:37:02 DEBUG ] 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG { 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG "data": { 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "result": [ 05:37:02 DEBUG { 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "pod": "ds-idrepo-0" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG "0" 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364222" 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364222" 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG { 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG "data": { 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "result": [ 05:37:02 DEBUG { 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "pod": "ds-idrepo-0" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG "447.79951226101855" 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364222" 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG { 05:37:02 DEBUG { 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG "data": { 05:37:02 DEBUG "data": { 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "result": [ 05:37:02 DEBUG "result": [ 05:37:02 DEBUG { 05:37:02 DEBUG { 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:37:02 DEBUG "pod": "ds-idrepo-0" 05:37:02 DEBUG }, 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:37:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:37:02 DEBUG }, 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG "14876635136" 05:37:02 DEBUG "0.0009275898627106422" 05:37:02 DEBUG ] 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 DEBUG ] 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364222" 05:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364222" 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG { 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG "data": { 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "result": [ 05:37:02 DEBUG { 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "instance": "10.142.0.11:9100" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG "instance": "10.142.0.51:9100" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG "96817.76666666669" 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG { 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG "data": { 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "result": [ 05:37:02 DEBUG { 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG "0.04141153657632651" 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:02 INFO [http_cmd]: http status code OK 05:37:02 DEBUG --- status code --- 05:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:02 DEBUG --- http response --- 05:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:37:02 DEBUG ----- output ----- 05:37:02 DEBUG { 05:37:02 DEBUG "status": "success", 05:37:02 DEBUG "data": { 05:37:02 DEBUG "resultType": "vector", 05:37:02 DEBUG "result": [ 05:37:02 DEBUG { 05:37:02 DEBUG "metric": { 05:37:02 DEBUG "instance": "10.142.0.11:9100" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG [...] 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG 05:37:02 DEBUG "instance": "10.142.0.51:9100" 05:37:02 DEBUG }, 05:37:02 DEBUG "value": [ 05:37:02 DEBUG 1735364222, 05:37:02 DEBUG "23483.733333333337" 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG ] 05:37:02 DEBUG } 05:37:02 DEBUG } 05:37:33 INFO 05:37:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:37:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:37:33 INFO [loop_until]: OK (rc = 0) 05:37:33 DEBUG --- stdout --- 05:37:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 19m 13375Mi rcs-687d95c5b-qs2xv 30m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:37:33 DEBUG --- stderr --- 05:37:33 DEBUG 05:37:35 INFO 05:37:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:37:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:37:35 INFO [loop_until]: OK (rc = 0) 05:37:35 DEBUG --- stdout --- 05:37:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 115m 0% 3450Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 98m 0% 18872Mi 66% 05:37:35 DEBUG --- stderr --- 05:37:35 DEBUG 05:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364281" 05:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364281" 05:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364281" 05:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364281" 05:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364281" 05:38:01 INFO [http_cmd]: http status code OK 05:38:01 INFO [http_cmd]: http status code OK 05:38:01 DEBUG --- status code --- 05:38:01 DEBUG --- status code --- 05:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:01 DEBUG --- http response --- 05:38:01 DEBUG --- http response --- 05:38:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:01 INFO [http_cmd]: http status code OK 05:38:01 DEBUG --- status code --- 05:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:01 DEBUG --- http response --- 05:38:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:01 INFO [http_cmd]: http status code OK 05:38:01 DEBUG --- status code --- 05:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:01 INFO [http_cmd]: http status code OK 05:38:01 DEBUG --- http response --- 05:38:01 DEBUG --- status code --- 05:38:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:01 DEBUG --- http response --- 05:38:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 ERROR [http_cmd]: ERROR 05:38:02 ERROR --- status code --- 05:38:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:38:02 ERROR --- http response --- 05:38:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:38:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG --- status code --- 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG --- status code --- 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364282" 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG --- status code --- 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG --- status code --- 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG --- http response --- 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:38:02 DEBUG { 05:38:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG { 05:38:02 DEBUG "data": { 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG "data": { 05:38:02 DEBUG "result": [ 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG { 05:38:02 DEBUG "result": [ 05:38:02 DEBUG "metric": { 05:38:02 DEBUG { 05:38:02 DEBUG "pod": "ds-idrepo-0" 05:38:02 DEBUG "metric": { 05:38:02 DEBUG }, 05:38:02 DEBUG "instance": "10.142.0.11:9100" 05:38:02 DEBUG "value": [ 05:38:02 DEBUG }, 05:38:02 DEBUG 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG [...] 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG [...] 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:38:02 DEBUG 05:38:02 DEBUG }, 05:38:02 DEBUG "instance": "10.142.0.51:9100" 05:38:02 DEBUG "value": [ 05:38:02 DEBUG }, 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG "831606784" 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG ] 05:38:02 DEBUG "0.0003666666666655753" 05:38:02 DEBUG } 05:38:02 DEBUG ] 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364282" 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG { 05:38:02 DEBUG { 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG "data": { 05:38:02 DEBUG "data": { 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG "result": [ 05:38:02 DEBUG "result": [ 05:38:02 DEBUG { 05:38:02 DEBUG { 05:38:02 DEBUG "metric": { 05:38:02 DEBUG "metric": { 05:38:02 DEBUG "instance": "10.142.0.11:9100" 05:38:02 DEBUG "instance": "10.142.0.11:9100" 05:38:02 DEBUG }, 05:38:02 DEBUG }, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG [...] 05:38:02 DEBUG [...] 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG "instance": "10.142.0.51:9100" 05:38:02 DEBUG "instance": "10.142.0.51:9100" 05:38:02 DEBUG }, 05:38:02 DEBUG }, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG "860575.3666666668" 05:38:02 DEBUG "0" 05:38:02 DEBUG ] 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 DEBUG ] 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG { 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG "data": { 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG "result": [ 05:38:02 DEBUG { 05:38:02 DEBUG "metric": { 05:38:02 DEBUG "pod": "ds-idrepo-0" 05:38:02 DEBUG }, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG [...] 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:38:02 DEBUG }, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG "0" 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG { 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG "data": { 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG "result": [ 05:38:02 DEBUG { 05:38:02 DEBUG "metric": { 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364282" 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364282" 05:38:02 DEBUG "pod": "ds-idrepo-0" 05:38:02 DEBUG }, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG [...] 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:38:02 DEBUG }, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG "981.5578189919447" 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364282" 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG { 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG "data": { 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364282" 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG "result": [ 05:38:02 DEBUG { 05:38:02 DEBUG "metric": { 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG "pod": "ds-idrepo-0" 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG }, 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG "value": [ 05:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364282" 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:38:02 DEBUG 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG 05:38:02 DEBUG { 05:38:02 DEBUG [...] 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG 05:38:02 DEBUG "data": { 05:38:02 DEBUG 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG 05:38:02 DEBUG "result": [ 05:38:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:38:02 DEBUG { 05:38:02 DEBUG }, 05:38:02 DEBUG "metric": { 05:38:02 DEBUG "value": [ 05:38:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG }, 05:38:02 DEBUG "0.0012884477396705508" 05:38:02 DEBUG "value": [ 05:38:02 DEBUG ] 05:38:02 DEBUG 05:38:02 DEBUG } 05:38:02 DEBUG 05:38:02 DEBUG ] 05:38:02 DEBUG 05:38:02 DEBUG } 05:38:02 DEBUG [...] 05:38:02 DEBUG } 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:38:02 DEBUG }, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG "14880497664" 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG { 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG "data": { 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG "result": [ 05:38:02 DEBUG { 05:38:02 DEBUG "metric": { 05:38:02 DEBUG "instance": "10.142.0.11:9100" 05:38:02 DEBUG }, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG [...] 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG "instance": "10.142.0.51:9100" 05:38:02 DEBUG }, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG "876807.1000000002" 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG --- http response --- 05:38:02 INFO [http_cmd]: http status code OK 05:38:02 DEBUG --- status code --- 05:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:38:02 DEBUG --- http response --- 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:38:02 DEBUG { 05:38:02 DEBUG ----- output ----- 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG { 05:38:02 DEBUG "data": { 05:38:02 DEBUG "status": "success", 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG "data": { 05:38:02 DEBUG "result": [ 05:38:02 DEBUG "resultType": "vector", 05:38:02 DEBUG { 05:38:02 DEBUG "result": [ 05:38:02 DEBUG "metric": { 05:38:02 DEBUG { 05:38:02 DEBUG "instance": "10.142.0.11:9100" 05:38:02 DEBUG "metric": { 05:38:02 DEBUG }, 05:38:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:38:02 DEBUG "value": [ 05:38:02 DEBUG }, 05:38:02 DEBUG 05:38:02 DEBUG "value": [ 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG [...] 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG [...] 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG 05:38:02 DEBUG "instance": "10.142.0.51:9100" 05:38:02 DEBUG 05:38:02 DEBUG }, 05:38:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:38:02 DEBUG "value": [ 05:38:02 DEBUG }, 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG "value": [ 05:38:02 DEBUG "23347.200000000004" 05:38:02 DEBUG 1735364282, 05:38:02 DEBUG ] 05:38:02 DEBUG "0.13506691414037164" 05:38:02 DEBUG } 05:38:02 DEBUG ] 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 DEBUG ] 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:02 DEBUG } 05:38:33 INFO 05:38:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:38:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:38:33 INFO [loop_until]: OK (rc = 0) 05:38:33 DEBUG --- stdout --- 05:38:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 66m 13375Mi rcs-687d95c5b-qs2xv 50m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:38:33 DEBUG --- stderr --- 05:38:33 DEBUG 05:38:35 INFO 05:38:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:38:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:38:35 INFO [loop_until]: OK (rc = 0) 05:38:35 DEBUG --- stdout --- 05:38:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 154m 0% 3451Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 151m 0% 18871Mi 66% 05:38:35 DEBUG --- stderr --- 05:38:35 DEBUG 05:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364341" 05:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364341" 05:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364341" 05:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364341" 05:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364341" 05:39:01 INFO [http_cmd]: http status code OK 05:39:01 DEBUG --- status code --- 05:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:01 DEBUG --- http response --- 05:39:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:01 INFO [http_cmd]: http status code OK 05:39:01 DEBUG --- status code --- 05:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:01 DEBUG --- http response --- 05:39:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:01 INFO [http_cmd]: http status code OK 05:39:01 DEBUG --- status code --- 05:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:01 DEBUG --- http response --- 05:39:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:01 INFO [http_cmd]: http status code OK 05:39:01 DEBUG --- status code --- 05:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:01 DEBUG --- http response --- 05:39:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:01 INFO [http_cmd]: http status code OK 05:39:01 DEBUG --- status code --- 05:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:01 DEBUG --- http response --- 05:39:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 ERROR [http_cmd]: ERROR 05:39:02 ERROR --- status code --- 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:39:02 ERROR --- http response --- 05:39:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:39:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364342" 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG --- status code --- 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG ----- output ----- 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG { 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG "data": { 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG "result": [ 05:39:02 DEBUG { 05:39:02 DEBUG { 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "metric": { 05:39:02 DEBUG "data": { 05:39:02 DEBUG "instance": "10.142.0.11:9100" 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG }, 05:39:02 DEBUG "result": [ 05:39:02 DEBUG "value": [ 05:39:02 DEBUG { 05:39:02 DEBUG 05:39:02 DEBUG "metric": { 05:39:02 DEBUG 05:39:02 DEBUG "pod": "ds-idrepo-0" 05:39:02 DEBUG 05:39:02 DEBUG }, 05:39:02 DEBUG [...] 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG "instance": "10.142.0.51:9100" 05:39:02 DEBUG [...] 05:39:02 DEBUG }, 05:39:02 DEBUG 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG 05:39:02 DEBUG "0.0011333333333368503" 05:39:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:39:02 DEBUG ] 05:39:02 DEBUG }, 05:39:02 DEBUG } 05:39:02 DEBUG "value": [ 05:39:02 DEBUG ] 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG } 05:39:02 DEBUG "831606784" 05:39:02 DEBUG } 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG { 05:39:02 DEBUG { 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "data": { 05:39:02 DEBUG "data": { 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG "result": [ 05:39:02 DEBUG "result": [ 05:39:02 DEBUG { 05:39:02 DEBUG { 05:39:02 DEBUG "metric": { 05:39:02 DEBUG "metric": { 05:39:02 DEBUG "instance": "10.142.0.11:9100" 05:39:02 DEBUG "instance": "10.142.0.11:9100" 05:39:02 DEBUG }, 05:39:02 DEBUG }, 05:39:02 DEBUG "value": [ 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG [...] 05:39:02 DEBUG [...] 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG "instance": "10.142.0.51:9100" 05:39:02 DEBUG "instance": "10.142.0.51:9100" 05:39:02 DEBUG }, 05:39:02 DEBUG }, 05:39:02 DEBUG "value": [ 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG "0.0010000000000000382" 05:39:02 DEBUG "1225671.4000000001" 05:39:02 DEBUG ] 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 DEBUG ] 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG { 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "data": { 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG "result": [ 05:39:02 DEBUG { 05:39:02 DEBUG "metric": { 05:39:02 DEBUG "pod": "ds-idrepo-0" 05:39:02 DEBUG }, 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG [...] 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:39:02 DEBUG }, 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG "0" 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG { 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "data": { 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG "result": [ 05:39:02 DEBUG { 05:39:02 DEBUG "metric": { 05:39:02 DEBUG "pod": "ds-idrepo-0" 05:39:02 DEBUG }, 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG [...] 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:39:02 DEBUG }, 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG "898.4523004239247" 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364342" 05:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364342" 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG { 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "data": { 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG "result": [ 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG { 05:39:02 DEBUG { 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "metric": { 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:39:02 DEBUG "data": { 05:39:02 DEBUG "instance": "10.142.0.11:9100" 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG }, 05:39:02 DEBUG { 05:39:02 DEBUG "result": [ 05:39:02 DEBUG "value": [ 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "data": { 05:39:02 DEBUG 05:39:02 DEBUG { 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG 05:39:02 DEBUG "metric": { 05:39:02 DEBUG "result": [ 05:39:02 DEBUG 05:39:02 DEBUG "pod": "ds-idrepo-0" 05:39:02 DEBUG { 05:39:02 DEBUG [...] 05:39:02 DEBUG }, 05:39:02 DEBUG "metric": { 05:39:02 DEBUG 05:39:02 DEBUG "value": [ 05:39:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG }, 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG "value": [ 05:39:02 DEBUG "instance": "10.142.0.51:9100" 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG }, 05:39:02 DEBUG [...] 05:39:02 DEBUG 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG 05:39:02 DEBUG [...] 05:39:02 DEBUG "1242379.3000000005" 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG ] 05:39:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:39:02 DEBUG 05:39:02 DEBUG } 05:39:02 DEBUG }, 05:39:02 DEBUG 05:39:02 DEBUG ] 05:39:02 DEBUG "value": [ 05:39:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:39:02 DEBUG } 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG }, 05:39:02 DEBUG } 05:39:02 DEBUG "0.0011748674088018028" 05:39:02 DEBUG "value": [ 05:39:02 DEBUG ] 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG } 05:39:02 DEBUG "14879416320" 05:39:02 DEBUG ] 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG --- status code --- 05:39:02 INFO [http_cmd]: http status code OK 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- status code --- 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:02 DEBUG --- http response --- 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG ----- output ----- 05:39:02 DEBUG { 05:39:02 DEBUG { 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "status": "success", 05:39:02 DEBUG "data": { 05:39:02 DEBUG "data": { 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG "resultType": "vector", 05:39:02 DEBUG "result": [ 05:39:02 DEBUG "result": [ 05:39:02 DEBUG { 05:39:02 DEBUG { 05:39:02 DEBUG "metric": { 05:39:02 DEBUG "metric": { 05:39:02 DEBUG "instance": "10.142.0.11:9100" 05:39:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:39:02 DEBUG }, 05:39:02 DEBUG }, 05:39:02 DEBUG "value": [ 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG [...] 05:39:02 DEBUG [...] 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG 05:39:02 DEBUG "instance": "10.142.0.51:9100" 05:39:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:39:02 DEBUG }, 05:39:02 DEBUG }, 05:39:02 DEBUG "value": [ 05:39:02 DEBUG "value": [ 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG 1735364342, 05:39:02 DEBUG "20753.06666666667" 05:39:02 DEBUG "0.0926056861499127" 05:39:02 DEBUG ] 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 DEBUG ] 05:39:02 DEBUG ] 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:02 DEBUG } 05:39:33 INFO 05:39:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:39:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:39:33 INFO [loop_until]: OK (rc = 0) 05:39:33 DEBUG --- stdout --- 05:39:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 101m 13376Mi rcs-687d95c5b-qs2xv 93m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:39:33 DEBUG --- stderr --- 05:39:33 DEBUG 05:39:35 INFO 05:39:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:39:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:39:35 INFO [loop_until]: OK (rc = 0) 05:39:35 DEBUG --- stdout --- 05:39:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 180m 1% 3454Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 147m 0% 18872Mi 66% 05:39:35 DEBUG --- stderr --- 05:39:35 DEBUG 05:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364401" 05:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364401" 05:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364401" 05:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364401" 05:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364401" 05:40:01 INFO [http_cmd]: http status code OK 05:40:01 DEBUG --- status code --- 05:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:01 DEBUG --- http response --- 05:40:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:01 INFO [http_cmd]: http status code OK 05:40:01 INFO [http_cmd]: http status code OK 05:40:01 INFO [http_cmd]: http status code OK 05:40:01 INFO [http_cmd]: http status code OK 05:40:01 DEBUG --- status code --- 05:40:01 DEBUG --- status code --- 05:40:01 DEBUG --- status code --- 05:40:01 DEBUG --- status code --- 05:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:01 DEBUG --- http response --- 05:40:01 DEBUG --- http response --- 05:40:01 DEBUG --- http response --- 05:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:01 DEBUG --- http response --- 05:40:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 ERROR [http_cmd]: ERROR 05:40:02 ERROR --- status code --- 05:40:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:40:02 ERROR --- http response --- 05:40:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:40:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364402" 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG --- status code --- 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG --- status code --- 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG --- status code --- 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG { 05:40:02 DEBUG { 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG "data": { 05:40:02 DEBUG "data": { 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "result": [ 05:40:02 DEBUG "result": [ 05:40:02 DEBUG { 05:40:02 DEBUG { 05:40:02 DEBUG "metric": { 05:40:02 DEBUG "metric": { 05:40:02 DEBUG "pod": "ds-idrepo-0" 05:40:02 DEBUG "instance": "10.142.0.11:9100" 05:40:02 DEBUG }, 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG [...] 05:40:02 DEBUG [...] 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:40:02 DEBUG "instance": "10.142.0.51:9100" 05:40:02 DEBUG }, 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG "831606784" 05:40:02 DEBUG "0.00043333333333445514" 05:40:02 DEBUG ] 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:02 DEBUG ] 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364402" 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG { 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG "data": { 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "result": [ 05:40:02 DEBUG { 05:40:02 DEBUG "metric": { 05:40:02 DEBUG "instance": "10.142.0.11:9100" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG [...] 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "instance": "10.142.0.51:9100" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG "0.0003333333333339776" 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG { 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG "data": { 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "result": [ 05:40:02 DEBUG { 05:40:02 DEBUG "metric": { 05:40:02 DEBUG "instance": "10.142.0.11:9100" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG [...] 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "instance": "10.142.0.51:9100" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG "2381725.4333333336" 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG { 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG "data": { 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "result": [ 05:40:02 DEBUG { 05:40:02 DEBUG "metric": { 05:40:02 DEBUG "pod": "ds-idrepo-0" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG [...] 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG "0" 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG { 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG "data": { 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "result": [ 05:40:02 DEBUG { 05:40:02 DEBUG "metric": { 05:40:02 DEBUG "pod": "ds-idrepo-0" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG [...] 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG "708.682299531576" 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364402" 05:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364402" 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG { 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG { 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG { 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG "data": { 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG "data": { 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "data": { 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "result": [ 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "result": [ 05:40:02 DEBUG { 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG "result": [ 05:40:02 DEBUG { 05:40:02 DEBUG "metric": { 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG { 05:40:02 DEBUG "metric": { 05:40:02 DEBUG "instance": "10.142.0.11:9100" 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG "metric": { 05:40:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:40:02 DEBUG }, 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:40:02 DEBUG "pod": "ds-idrepo-0" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 05:40:02 DEBUG { 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "data": { 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG [...] 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG 05:40:02 DEBUG [...] 05:40:02 DEBUG 05:40:02 DEBUG "result": [ 05:40:02 DEBUG [...] 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG { 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "metric": { 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "instance": "10.142.0.51:9100" 05:40:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:40:02 DEBUG 05:40:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:40:02 DEBUG }, 05:40:02 DEBUG }, 05:40:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG }, 05:40:02 INFO [http_cmd]: http status code OK 05:40:02 DEBUG "value": [ 05:40:02 DEBUG "14877728768" 05:40:02 DEBUG 05:40:02 DEBUG "value": [ 05:40:02 DEBUG --- status code --- 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG ] 05:40:02 DEBUG 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:02 DEBUG "2398940.700000001" 05:40:02 DEBUG } 05:40:02 DEBUG 05:40:02 DEBUG "0.0012061730276360155" 05:40:02 DEBUG --- http response --- 05:40:02 DEBUG ] 05:40:02 DEBUG ] 05:40:02 DEBUG [...] 05:40:02 DEBUG ] 05:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:02 DEBUG 05:40:02 DEBUG } 05:40:02 DEBUG ----- output ----- 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG 05:40:02 DEBUG ] 05:40:02 DEBUG { 05:40:02 DEBUG } 05:40:02 DEBUG 05:40:02 DEBUG } 05:40:02 DEBUG "status": "success", 05:40:02 DEBUG } 05:40:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:40:02 DEBUG } 05:40:02 DEBUG "data": { 05:40:02 DEBUG }, 05:40:02 DEBUG "resultType": "vector", 05:40:02 DEBUG "value": [ 05:40:02 DEBUG "result": [ 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG { 05:40:02 DEBUG "0.1591976914806791" 05:40:02 DEBUG "metric": { 05:40:02 DEBUG ] 05:40:02 DEBUG "instance": "10.142.0.11:9100" 05:40:02 DEBUG } 05:40:02 DEBUG }, 05:40:02 DEBUG ] 05:40:02 DEBUG "value": [ 05:40:02 DEBUG } 05:40:02 DEBUG 05:40:02 DEBUG } 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG [...] 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG 05:40:02 DEBUG "instance": "10.142.0.51:9100" 05:40:02 DEBUG }, 05:40:02 DEBUG "value": [ 05:40:02 DEBUG 1735364402, 05:40:02 DEBUG "28262.400000000005" 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG ] 05:40:02 DEBUG } 05:40:02 DEBUG } 05:40:33 INFO 05:40:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:40:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:40:33 INFO [loop_until]: OK (rc = 0) 05:40:33 DEBUG --- stdout --- 05:40:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 132m 13375Mi rcs-687d95c5b-qs2xv 133m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:40:33 DEBUG --- stderr --- 05:40:33 DEBUG 05:40:35 INFO 05:40:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:40:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:40:35 INFO [loop_until]: OK (rc = 0) 05:40:35 DEBUG --- stdout --- 05:40:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 238m 1% 3439Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 216m 0% 18874Mi 66% 05:40:35 DEBUG --- stderr --- 05:40:35 DEBUG 05:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364461" 05:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364461" 05:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364461" 05:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364461" 05:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364461" 05:41:01 INFO [http_cmd]: http status code OK 05:41:01 DEBUG --- status code --- 05:41:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:01 DEBUG --- http response --- 05:41:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:01 INFO [http_cmd]: http status code OK 05:41:01 INFO [http_cmd]: http status code OK 05:41:01 DEBUG --- status code --- 05:41:01 DEBUG --- status code --- 05:41:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:01 DEBUG --- http response --- 05:41:01 DEBUG --- http response --- 05:41:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:01 INFO [http_cmd]: http status code OK 05:41:01 INFO [http_cmd]: http status code OK 05:41:01 DEBUG --- status code --- 05:41:01 DEBUG --- status code --- 05:41:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:01 DEBUG --- http response --- 05:41:01 DEBUG --- http response --- 05:41:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 ERROR [http_cmd]: ERROR 05:41:02 DEBUG --- status code --- 05:41:02 ERROR --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 ERROR --- http response --- 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:41:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364462" 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- status code --- 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG --- status code --- 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- status code --- 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { 05:41:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:02 DEBUG { 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG "data": { 05:41:02 DEBUG "data": { 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG "result": [ 05:41:02 DEBUG "result": [ 05:41:02 DEBUG { 05:41:02 DEBUG { 05:41:02 DEBUG "metric": { 05:41:02 DEBUG "metric": { 05:41:02 DEBUG "pod": "ds-idrepo-0" 05:41:02 DEBUG "instance": "10.142.0.11:9100" 05:41:02 DEBUG }, 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG [...] 05:41:02 DEBUG [...] 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:41:02 DEBUG "instance": "10.142.0.51:9100" 05:41:02 DEBUG }, 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG "831672320" 05:41:02 DEBUG "0.0005999999999933912" 05:41:02 DEBUG ] 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364462" 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG { 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG "data": { 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG "result": [ 05:41:02 DEBUG { 05:41:02 DEBUG "metric": { 05:41:02 DEBUG "instance": "10.142.0.11:9100" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG [...] 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG "instance": "10.142.0.51:9100" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG "0" 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG { 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG "data": { 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG "result": [ 05:41:02 DEBUG { 05:41:02 DEBUG "metric": { 05:41:02 DEBUG "instance": "10.142.0.11:9100" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG [...] 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG "instance": "10.142.0.51:9100" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG "2376556.8666666667" 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG { 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG "data": { 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG "result": [ 05:41:02 DEBUG { 05:41:02 DEBUG "metric": { 05:41:02 DEBUG "pod": "ds-idrepo-0" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG [...] 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG "0" 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG { 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG "data": { 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG "result": [ 05:41:02 DEBUG { 05:41:02 DEBUG "metric": { 05:41:02 DEBUG "pod": "ds-idrepo-0" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG [...] 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG "921.2102387703195" 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364462" 05:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364462" 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG { 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG "data": { 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG "result": [ 05:41:02 DEBUG { 05:41:02 DEBUG { 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG "metric": { 05:41:02 DEBUG "data": { 05:41:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG }, 05:41:02 DEBUG "result": [ 05:41:02 DEBUG "value": [ 05:41:02 DEBUG { 05:41:02 DEBUG 05:41:02 DEBUG "metric": { 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG 05:41:02 DEBUG "instance": "10.142.0.11:9100" 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG 05:41:02 DEBUG }, 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG [...] 05:41:02 DEBUG "value": [ 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG { 05:41:02 DEBUG { 05:41:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:41:02 DEBUG [...] 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG }, 05:41:02 DEBUG 05:41:02 DEBUG "data": { 05:41:02 DEBUG "data": { 05:41:02 INFO [http_cmd]: http status code OK 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG "result": [ 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG 05:41:02 DEBUG "result": [ 05:41:02 DEBUG --- status code --- 05:41:02 DEBUG { 05:41:02 DEBUG "14880575488" 05:41:02 DEBUG "instance": "10.142.0.51:9100" 05:41:02 DEBUG { 05:41:02 DEBUG "metric": { 05:41:02 DEBUG "metric": { 05:41:02 DEBUG ] 05:41:02 DEBUG }, 05:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:02 DEBUG "instance": "10.142.0.11:9100" 05:41:02 DEBUG "pod": "ds-idrepo-0" 05:41:02 DEBUG } 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG }, 05:41:02 DEBUG }, 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG "2391579.1000000006" 05:41:02 DEBUG "value": [ 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG 05:41:02 DEBUG --- http response --- 05:41:02 DEBUG 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 DEBUG 05:41:02 DEBUG [...] 05:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:41:02 DEBUG [...] 05:41:02 DEBUG 05:41:02 DEBUG ----- output ----- 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG { 05:41:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:41:02 DEBUG 05:41:02 DEBUG "status": "success", 05:41:02 DEBUG }, 05:41:02 DEBUG "instance": "10.142.0.51:9100" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG "data": { 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG "0.0012370505694674168" 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG "resultType": "vector", 05:41:02 DEBUG ] 05:41:02 DEBUG "19797.333333333336" 05:41:02 DEBUG "result": [ 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG { 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:02 DEBUG "metric": { 05:41:02 DEBUG } 05:41:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG [...] 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG 05:41:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:41:02 DEBUG }, 05:41:02 DEBUG "value": [ 05:41:02 DEBUG 1735364462, 05:41:02 DEBUG "0.1359147178733211" 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG ] 05:41:02 DEBUG } 05:41:02 DEBUG } 05:41:33 INFO 05:41:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:41:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:41:33 INFO [loop_until]: OK (rc = 0) 05:41:33 DEBUG --- stdout --- 05:41:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 130m 13376Mi rcs-687d95c5b-qs2xv 112m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:41:33 DEBUG --- stderr --- 05:41:33 DEBUG 05:41:35 INFO 05:41:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:41:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:41:35 INFO [loop_until]: OK (rc = 0) 05:41:35 DEBUG --- stdout --- 05:41:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 235m 1% 3449Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 199m 0% 18873Mi 66% 05:41:35 DEBUG --- stderr --- 05:41:35 DEBUG 05:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364521" 05:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364521" 05:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364521" 05:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364521" 05:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364521" 05:42:01 INFO [http_cmd]: http status code OK 05:42:01 DEBUG --- status code --- 05:42:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:01 DEBUG --- http response --- 05:42:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:01 INFO [http_cmd]: http status code OK 05:42:01 INFO [http_cmd]: http status code OK 05:42:01 DEBUG --- status code --- 05:42:01 INFO [http_cmd]: http status code OK 05:42:01 DEBUG --- status code --- 05:42:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:01 DEBUG --- status code --- 05:42:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:01 DEBUG --- http response --- 05:42:01 INFO [http_cmd]: http status code OK 05:42:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:01 DEBUG --- http response --- 05:42:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:01 DEBUG --- status code --- 05:42:01 DEBUG --- http response --- 05:42:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:01 DEBUG --- http response --- 05:42:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 ERROR [http_cmd]: ERROR 05:42:02 DEBUG --- http response --- 05:42:02 ERROR --- status code --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:42:02 ERROR --- http response --- 05:42:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:42:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 DEBUG --- http response --- 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 DEBUG --- http response --- 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG --- status code --- 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364522" 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG --- status code --- 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364522" 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG --- http response --- 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG { 05:42:02 DEBUG { 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:42:02 DEBUG { 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG "data": { 05:42:02 DEBUG "data": { 05:42:02 DEBUG { 05:42:02 DEBUG "data": { 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG "result": [ 05:42:02 DEBUG "result": [ 05:42:02 DEBUG "data": { 05:42:02 DEBUG "result": [ 05:42:02 DEBUG { 05:42:02 DEBUG { 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG { 05:42:02 DEBUG "metric": { 05:42:02 DEBUG "metric": { 05:42:02 DEBUG "result": [ 05:42:02 DEBUG "metric": { 05:42:02 DEBUG "pod": "ds-idrepo-0" 05:42:02 DEBUG "instance": "10.142.0.11:9100" 05:42:02 DEBUG { 05:42:02 DEBUG "instance": "10.142.0.11:9100" 05:42:02 DEBUG }, 05:42:02 DEBUG }, 05:42:02 DEBUG "metric": { 05:42:02 DEBUG }, 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "instance": "10.142.0.11:9100" 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG }, 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG [...] 05:42:02 DEBUG [...] 05:42:02 DEBUG 05:42:02 DEBUG [...] 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG [...] 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:42:02 DEBUG "instance": "10.142.0.51:9100" 05:42:02 DEBUG 05:42:02 DEBUG "instance": "10.142.0.51:9100" 05:42:02 DEBUG }, 05:42:02 DEBUG }, 05:42:02 DEBUG 05:42:02 DEBUG }, 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "instance": "10.142.0.51:9100" 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG }, 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG "831672320" 05:42:02 DEBUG "0.0013000000000033653" 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "0.0020000000000000764" 05:42:02 DEBUG ] 05:42:02 DEBUG ] 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG ] 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG "1134401.8666666667" 05:42:02 DEBUG } 05:42:02 DEBUG ] 05:42:02 DEBUG ] 05:42:02 DEBUG ] 05:42:02 DEBUG ] 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG ] 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG { 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG "data": { 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG "result": [ 05:42:02 DEBUG { 05:42:02 DEBUG "metric": { 05:42:02 DEBUG "pod": "ds-idrepo-0" 05:42:02 DEBUG }, 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG [...] 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:42:02 DEBUG }, 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG "0" 05:42:02 DEBUG ] 05:42:02 DEBUG } 05:42:02 DEBUG ] 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG { 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG "data": { 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG "result": [ 05:42:02 DEBUG { 05:42:02 DEBUG "metric": { 05:42:02 DEBUG "pod": "ds-idrepo-0" 05:42:02 DEBUG }, 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG [...] 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:42:02 DEBUG }, 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG "778.1929626543046" 05:42:02 DEBUG ] 05:42:02 DEBUG } 05:42:02 DEBUG ] 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364522" 05:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364522" 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG { 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG "data": { 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG "result": [ 05:42:02 DEBUG { 05:42:02 DEBUG "metric": { 05:42:02 DEBUG "instance": "10.142.0.11:9100" 05:42:02 DEBUG }, 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG [...] 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG 05:42:02 DEBUG --- http response --- 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG "instance": "10.142.0.51:9100" 05:42:02 DEBUG ----- output ----- 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG }, 05:42:02 INFO [http_cmd]: http status code OK 05:42:02 DEBUG { 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG "value": [ 05:42:02 DEBUG --- status code --- 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:02 DEBUG "data": { 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG "1152008.2333333336" 05:42:02 DEBUG --- http response --- 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG { 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:42:02 DEBUG ] 05:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:42:02 DEBUG "result": [ 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG } 05:42:02 DEBUG ----- output ----- 05:42:02 DEBUG { 05:42:02 DEBUG "data": { 05:42:02 DEBUG { 05:42:02 DEBUG ] 05:42:02 DEBUG { 05:42:02 DEBUG "metric": { 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG } 05:42:02 DEBUG "status": "success", 05:42:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:42:02 DEBUG "result": [ 05:42:02 DEBUG "data": { 05:42:02 DEBUG } 05:42:02 DEBUG "data": { 05:42:02 DEBUG }, 05:42:02 DEBUG { 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG "resultType": "vector", 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "metric": { 05:42:02 DEBUG "result": [ 05:42:02 DEBUG "result": [ 05:42:02 DEBUG 05:42:02 DEBUG "pod": "ds-idrepo-0" 05:42:02 DEBUG { 05:42:02 DEBUG { 05:42:02 DEBUG 05:42:02 DEBUG }, 05:42:02 DEBUG "metric": { 05:42:02 DEBUG "metric": { 05:42:02 DEBUG 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "instance": "10.142.0.11:9100" 05:42:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:42:02 DEBUG [...] 05:42:02 DEBUG 05:42:02 DEBUG }, 05:42:02 DEBUG }, 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG [...] 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG }, 05:42:02 DEBUG 05:42:02 DEBUG [...] 05:42:02 DEBUG [...] 05:42:02 DEBUG "value": [ 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG "14880825344" 05:42:02 DEBUG }, 05:42:02 DEBUG 05:42:02 DEBUG 05:42:02 DEBUG ] 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "instance": "10.142.0.51:9100" 05:42:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:42:02 DEBUG } 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG }, 05:42:02 DEBUG }, 05:42:02 DEBUG ] 05:42:02 DEBUG "0.001265205126909592" 05:42:02 DEBUG "value": [ 05:42:02 DEBUG "value": [ 05:42:02 DEBUG } 05:42:02 DEBUG ] 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG 1735364522, 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG "20343.46666666667" 05:42:02 DEBUG "0.13538510630490583" 05:42:02 DEBUG ] 05:42:02 DEBUG ] 05:42:02 DEBUG ] 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG ] 05:42:02 DEBUG ] 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:02 DEBUG } 05:42:33 INFO 05:42:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:42:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:42:33 INFO [loop_until]: OK (rc = 0) 05:42:33 DEBUG --- stdout --- 05:42:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 600m 13376Mi rcs-687d95c5b-qs2xv 44m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:42:33 DEBUG --- stderr --- 05:42:33 DEBUG 05:42:35 INFO 05:42:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:42:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:42:35 INFO [loop_until]: OK (rc = 0) 05:42:35 DEBUG --- stdout --- 05:42:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 159m 1% 3450Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 670m 2% 18874Mi 66% 05:42:35 DEBUG --- stderr --- 05:42:35 DEBUG 05:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364581" 05:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364581" 05:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364581" 05:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364581" 05:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364581" 05:43:01 INFO [http_cmd]: http status code OK 05:43:01 DEBUG --- status code --- 05:43:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:01 DEBUG --- http response --- 05:43:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:01 INFO [http_cmd]: http status code OK 05:43:01 DEBUG --- status code --- 05:43:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:01 DEBUG --- http response --- 05:43:01 INFO [http_cmd]: http status code OK 05:43:01 DEBUG --- status code --- 05:43:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:01 DEBUG --- http response --- 05:43:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:01 INFO [http_cmd]: http status code OK 05:43:01 DEBUG --- status code --- 05:43:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:01 DEBUG --- http response --- 05:43:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:01 INFO [http_cmd]: http status code OK 05:43:01 DEBUG --- status code --- 05:43:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:01 DEBUG --- http response --- 05:43:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 ERROR [http_cmd]: ERROR 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- http response --- 05:43:02 ERROR --- status code --- 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 ERROR --- http response --- 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 DEBUG --- status code --- 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364582" 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG --- status code --- 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { 05:43:02 DEBUG { 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG "data": { 05:43:02 DEBUG "data": { 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG { 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG { 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG "result": [ 05:43:02 DEBUG "result": [ 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG "data": { 05:43:02 DEBUG { 05:43:02 DEBUG { 05:43:02 DEBUG "data": { 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG "result": [ 05:43:02 DEBUG "instance": "10.142.0.11:9100" 05:43:02 DEBUG "instance": "10.142.0.11:9100" 05:43:02 DEBUG "result": [ 05:43:02 DEBUG { 05:43:02 DEBUG }, 05:43:02 DEBUG }, 05:43:02 DEBUG { 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "value": [ 05:43:02 DEBUG "value": [ 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "pod": "ds-idrepo-0" 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG "instance": "10.142.0.11:9100" 05:43:02 DEBUG }, 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG }, 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG [...] 05:43:02 DEBUG 05:43:02 DEBUG [...] 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG [...] 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG [...] 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG "instance": "10.142.0.51:9100" 05:43:02 DEBUG 05:43:02 DEBUG "instance": "10.142.0.51:9100" 05:43:02 DEBUG 05:43:02 DEBUG }, 05:43:02 DEBUG 05:43:02 DEBUG }, 05:43:02 DEBUG "instance": "10.142.0.51:9100" 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 05:43:02 DEBUG "value": [ 05:43:02 DEBUG }, 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG "value": [ 05:43:02 DEBUG "76103.76666666669" 05:43:02 DEBUG }, 05:43:02 DEBUG "0.0005666666666646356" 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG ] 05:43:02 DEBUG "0" 05:43:02 DEBUG ] 05:43:02 DEBUG "value": [ 05:43:02 DEBUG } 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 DEBUG "831680512" 05:43:02 DEBUG } 05:43:02 DEBUG ] 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG { 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG "data": { 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG "result": [ 05:43:02 DEBUG { 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "pod": "ds-idrepo-0" 05:43:02 DEBUG }, 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG [...] 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:43:02 DEBUG }, 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG "0" 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG { 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG "data": { 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG "result": [ 05:43:02 DEBUG { 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "pod": "ds-idrepo-0" 05:43:02 DEBUG }, 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG [...] 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:43:02 DEBUG }, 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG "1122.6846354221564" 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364582" 05:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364582" 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG { 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG "data": { 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG "result": [ 05:43:02 DEBUG { 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "instance": "10.142.0.11:9100" 05:43:02 DEBUG }, 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG [...] 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG "instance": "10.142.0.51:9100" 05:43:02 DEBUG }, 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG "93674.46666666667" 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG { 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG "data": { 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG "result": [ 05:43:02 DEBUG { 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:43:02 DEBUG }, 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 INFO [http_cmd]: http status code OK 05:43:02 DEBUG [...] 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG --- status code --- 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:43:02 DEBUG 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG --- http response --- 05:43:02 DEBUG { 05:43:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG }, 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG ----- output ----- 05:43:02 DEBUG "data": { 05:43:02 DEBUG "value": [ 05:43:02 DEBUG { 05:43:02 DEBUG { 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG "status": "success", 05:43:02 DEBUG "result": [ 05:43:02 DEBUG "14883409920" 05:43:02 DEBUG "data": { 05:43:02 DEBUG "data": { 05:43:02 DEBUG { 05:43:02 DEBUG ] 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG "resultType": "vector", 05:43:02 DEBUG "metric": { 05:43:02 DEBUG } 05:43:02 DEBUG "result": [ 05:43:02 DEBUG "result": [ 05:43:02 DEBUG "instance": "10.142.0.11:9100" 05:43:02 DEBUG ] 05:43:02 DEBUG { 05:43:02 DEBUG { 05:43:02 DEBUG }, 05:43:02 DEBUG } 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "metric": { 05:43:02 DEBUG "value": [ 05:43:02 DEBUG } 05:43:02 DEBUG "pod": "ds-idrepo-0" 05:43:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:43:02 DEBUG 05:43:02 DEBUG }, 05:43:02 DEBUG }, 05:43:02 DEBUG 05:43:02 DEBUG "value": [ 05:43:02 DEBUG "value": [ 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG [...] 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG [...] 05:43:02 DEBUG [...] 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG "instance": "10.142.0.51:9100" 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG }, 05:43:02 DEBUG 05:43:02 DEBUG 05:43:02 DEBUG "value": [ 05:43:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:43:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG }, 05:43:02 DEBUG }, 05:43:02 DEBUG "24712.533333333336" 05:43:02 DEBUG "value": [ 05:43:02 DEBUG "value": [ 05:43:02 DEBUG ] 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG 1735364582, 05:43:02 DEBUG } 05:43:02 DEBUG "0.0011441360008116448" 05:43:02 DEBUG "0.038189061324409045" 05:43:02 DEBUG ] 05:43:02 DEBUG ] 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 DEBUG ] 05:43:02 DEBUG ] 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:02 DEBUG } 05:43:33 INFO 05:43:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:43:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:43:33 INFO [loop_until]: OK (rc = 0) 05:43:33 DEBUG --- stdout --- 05:43:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 6m 13377Mi rcs-687d95c5b-qs2xv 54m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:43:33 DEBUG --- stderr --- 05:43:33 DEBUG 05:43:35 INFO 05:43:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:43:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:43:35 INFO [loop_until]: OK (rc = 0) 05:43:35 DEBUG --- stdout --- 05:43:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 120m 0% 3395Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 95m 0% 18863Mi 66% 05:43:35 DEBUG --- stderr --- 05:43:35 DEBUG 05:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364641" 05:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364641" 05:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364641" 05:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364641" 05:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364641" 05:44:01 INFO [http_cmd]: http status code OK 05:44:01 INFO [http_cmd]: http status code OK 05:44:01 INFO [http_cmd]: http status code OK 05:44:01 DEBUG --- status code --- 05:44:01 INFO [http_cmd]: http status code OK 05:44:01 INFO [http_cmd]: http status code OK 05:44:01 DEBUG --- status code --- 05:44:01 DEBUG --- status code --- 05:44:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:01 DEBUG --- status code --- 05:44:01 DEBUG --- status code --- 05:44:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:01 DEBUG --- http response --- 05:44:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:01 DEBUG --- http response --- 05:44:01 DEBUG --- http response --- 05:44:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:01 DEBUG --- http response --- 05:44:01 DEBUG --- http response --- 05:44:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 ERROR [http_cmd]: ERROR 05:44:02 ERROR --- status code --- 05:44:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 ERROR --- http response --- 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG { 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG "data": { 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG "result": [ 05:44:02 DEBUG { 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "instance": "10.142.0.11:9100" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- status code --- 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG "instance": "10.142.0.51:9100" 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG }, 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG "value": [ 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:44:02 DEBUG "1544882.0000000002" 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG ] 05:44:02 DEBUG { 05:44:02 DEBUG "data": { 05:44:02 DEBUG { 05:44:02 DEBUG } 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG ] 05:44:02 DEBUG "data": { 05:44:02 DEBUG "result": [ 05:44:02 DEBUG "data": { 05:44:02 DEBUG } 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG { 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG } 05:44:02 DEBUG "result": [ 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "result": [ 05:44:02 DEBUG { 05:44:02 DEBUG "instance": "10.142.0.11:9100" 05:44:02 DEBUG { 05:44:02 DEBUG "metric": { 05:44:02 DEBUG }, 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "pod": "ds-idrepo-0" 05:44:02 DEBUG "value": [ 05:44:02 DEBUG "instance": "10.142.0.11:9100" 05:44:02 DEBUG }, 05:44:02 DEBUG 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG "instance": "10.142.0.51:9100" 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG }, 05:44:02 DEBUG 05:44:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:44:02 DEBUG "value": [ 05:44:02 DEBUG "instance": "10.142.0.51:9100" 05:44:02 DEBUG }, 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG "0.00036666666666936487" 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG ] 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG "831672320" 05:44:02 DEBUG } 05:44:02 DEBUG "0" 05:44:02 DEBUG ] 05:44:02 DEBUG ] 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG { 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG "data": { 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG "result": [ 05:44:02 DEBUG { 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "pod": "ds-idrepo-0" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG "0" 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG { 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG "data": { 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG "result": [ 05:44:02 DEBUG { 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "pod": "ds-idrepo-0" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG "448.1073590547735" 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG { 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG "data": { 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG "result": [ 05:44:02 DEBUG { 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "instance": "10.142.0.11:9100" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG "instance": "10.142.0.51:9100" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG "1561629.3000000005" 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364642" 05:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364642" 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG { 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG "data": { 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG "result": [ 05:44:02 DEBUG { 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG "14878642176" 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG { 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG "data": { 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG "result": [ 05:44:02 DEBUG { 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "instance": "10.142.0.11:9100" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG "instance": "10.142.0.51:9100" 05:44:02 DEBUG }, 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG "6690.133333333334" 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 INFO [http_cmd]: http status code OK 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG --- status code --- 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG --- http response --- 05:44:02 DEBUG { 05:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG ----- output ----- 05:44:02 DEBUG "data": { 05:44:02 DEBUG { 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG "status": "success", 05:44:02 DEBUG "result": [ 05:44:02 DEBUG "data": { 05:44:02 DEBUG { 05:44:02 DEBUG "resultType": "vector", 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "result": [ 05:44:02 DEBUG "pod": "ds-idrepo-0" 05:44:02 DEBUG { 05:44:02 DEBUG }, 05:44:02 DEBUG "metric": { 05:44:02 DEBUG "value": [ 05:44:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:44:02 DEBUG 05:44:02 DEBUG }, 05:44:02 DEBUG 05:44:02 DEBUG "value": [ 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG [...] 05:44:02 DEBUG 05:44:02 DEBUG 05:44:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:44:02 DEBUG 05:44:02 DEBUG }, 05:44:02 DEBUG 05:44:02 DEBUG "value": [ 05:44:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG }, 05:44:02 DEBUG "0.001301202446265498" 05:44:02 DEBUG "value": [ 05:44:02 DEBUG ] 05:44:02 DEBUG 1735364642, 05:44:02 DEBUG } 05:44:02 DEBUG "0.17408715621043222" 05:44:02 DEBUG ] 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:02 DEBUG ] 05:44:02 DEBUG } 05:44:02 DEBUG } 05:44:33 INFO 05:44:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:44:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:44:33 INFO [loop_until]: OK (rc = 0) 05:44:33 DEBUG --- stdout --- 05:44:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 23m 13378Mi rcs-687d95c5b-qs2xv 82m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:44:33 DEBUG --- stderr --- 05:44:33 DEBUG 05:44:35 INFO 05:44:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:44:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:44:35 INFO [loop_until]: OK (rc = 0) 05:44:35 DEBUG --- stdout --- 05:44:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 213m 1% 3412Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 166m 0% 18865Mi 66% 05:44:35 DEBUG --- stderr --- 05:44:35 DEBUG 05:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364701" 05:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364701" 05:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364701" 05:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364701" 05:45:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364701" 05:45:01 INFO [http_cmd]: http status code OK 05:45:01 DEBUG --- status code --- 05:45:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:01 DEBUG --- http response --- 05:45:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:01 INFO [http_cmd]: http status code OK 05:45:01 DEBUG --- status code --- 05:45:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:01 DEBUG --- http response --- 05:45:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:01 INFO [http_cmd]: http status code OK 05:45:01 INFO [http_cmd]: http status code OK 05:45:01 INFO [http_cmd]: http status code OK 05:45:01 DEBUG --- status code --- 05:45:01 DEBUG --- status code --- 05:45:01 DEBUG --- status code --- 05:45:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:01 DEBUG --- http response --- 05:45:01 DEBUG --- http response --- 05:45:01 DEBUG --- http response --- 05:45:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 ERROR [http_cmd]: ERROR 05:45:02 DEBUG --- http response --- 05:45:02 ERROR --- status code --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:45:02 ERROR --- http response --- 05:45:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:45:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 DEBUG --- status code --- 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364702" 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG --- status code --- 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364702" 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364702" 05:45:02 DEBUG --- http response --- 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG --- status code --- 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- http response --- 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:45:02 DEBUG { 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG { 05:45:02 DEBUG { 05:45:02 DEBUG { 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:45:02 DEBUG "data": { 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "data": { 05:45:02 DEBUG "data": { 05:45:02 DEBUG "data": { 05:45:02 DEBUG { 05:45:02 DEBUG "result": [ 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG { 05:45:02 DEBUG "result": [ 05:45:02 DEBUG "result": [ 05:45:02 DEBUG "result": [ 05:45:02 DEBUG "data": { 05:45:02 DEBUG "metric": { 05:45:02 DEBUG { 05:45:02 DEBUG { 05:45:02 DEBUG { 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "instance": "10.142.0.11:9100" 05:45:02 DEBUG "metric": { 05:45:02 DEBUG "metric": { 05:45:02 DEBUG "metric": { 05:45:02 DEBUG "result": [ 05:45:02 DEBUG }, 05:45:02 DEBUG "pod": "ds-idrepo-0" 05:45:02 DEBUG "instance": "10.142.0.11:9100" 05:45:02 DEBUG "instance": "10.142.0.11:9100" 05:45:02 DEBUG { 05:45:02 DEBUG "value": [ 05:45:02 DEBUG }, 05:45:02 DEBUG }, 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 05:45:02 DEBUG "value": [ 05:45:02 DEBUG "value": [ 05:45:02 DEBUG "metric": { 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG "pod": "ds-idrepo-0" 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG }, 05:45:02 DEBUG 05:45:02 DEBUG [...] 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG "value": [ 05:45:02 DEBUG [...] 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 DEBUG 05:45:02 DEBUG [...] 05:45:02 DEBUG [...] 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG "instance": "10.142.0.51:9100" 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG [...] 05:45:02 DEBUG "instance": "10.142.0.51:9100" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG "instance": "10.142.0.51:9100" 05:45:02 DEBUG 05:45:02 DEBUG }, 05:45:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG }, 05:45:02 DEBUG 05:45:02 DEBUG "value": [ 05:45:02 DEBUG }, 05:45:02 DEBUG "1878260.1000000006" 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG ] 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:45:02 DEBUG "0" 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG } 05:45:02 DEBUG "0.0005666666666722147" 05:45:02 DEBUG }, 05:45:02 DEBUG ] 05:45:02 DEBUG "831672320" 05:45:02 DEBUG ] 05:45:02 DEBUG ] 05:45:02 DEBUG "value": [ 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG "0" 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG { 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG "data": { 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "result": [ 05:45:02 DEBUG { 05:45:02 DEBUG "metric": { 05:45:02 DEBUG "pod": "ds-idrepo-0" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG [...] 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG "753.3363738310948" 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364702" 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG { 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG "data": { 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "result": [ 05:45:02 DEBUG { 05:45:02 DEBUG "metric": { 05:45:02 DEBUG "instance": "10.142.0.11:9100" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG [...] 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364702" 05:45:02 DEBUG 05:45:02 DEBUG "instance": "10.142.0.51:9100" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG "1894886.2333333336" 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364702" 05:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364702" 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG { 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG "data": { 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "result": [ 05:45:02 DEBUG { 05:45:02 DEBUG "metric": { 05:45:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG [...] 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG "14876749824" 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG { 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG "data": { 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "result": [ 05:45:02 DEBUG { 05:45:02 DEBUG "metric": { 05:45:02 DEBUG "instance": "10.142.0.11:9100" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG [...] 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG "instance": "10.142.0.51:9100" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG "27989.333333333336" 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG { 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG "data": { 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "result": [ 05:45:02 DEBUG { 05:45:02 DEBUG "metric": { 05:45:02 DEBUG "pod": "ds-idrepo-0" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 05:45:02 INFO [http_cmd]: http status code OK 05:45:02 DEBUG 05:45:02 DEBUG --- status code --- 05:45:02 DEBUG 05:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:02 DEBUG [...] 05:45:02 DEBUG --- http response --- 05:45:02 DEBUG 05:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:45:02 DEBUG 05:45:02 DEBUG ----- output ----- 05:45:02 DEBUG 05:45:02 DEBUG { 05:45:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:45:02 DEBUG "status": "success", 05:45:02 DEBUG }, 05:45:02 DEBUG "data": { 05:45:02 DEBUG "value": [ 05:45:02 DEBUG "resultType": "vector", 05:45:02 DEBUG "result": [ 05:45:02 DEBUG { 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG "metric": { 05:45:02 DEBUG "0.0013192698077009535" 05:45:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:45:02 DEBUG ] 05:45:02 DEBUG }, 05:45:02 DEBUG } 05:45:02 DEBUG "value": [ 05:45:02 DEBUG ] 05:45:02 DEBUG 05:45:02 DEBUG } 05:45:02 DEBUG 05:45:02 DEBUG } 05:45:02 DEBUG 05:45:02 DEBUG [...] 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG 05:45:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:45:02 DEBUG }, 05:45:02 DEBUG "value": [ 05:45:02 DEBUG 1735364702, 05:45:02 DEBUG "0.15372936219080693" 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG ] 05:45:02 DEBUG } 05:45:02 DEBUG } 05:45:33 INFO 05:45:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:45:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:45:33 INFO [loop_until]: OK (rc = 0) 05:45:33 DEBUG --- stdout --- 05:45:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 98m 13377Mi rcs-687d95c5b-qs2xv 111m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:45:33 DEBUG --- stderr --- 05:45:33 DEBUG 05:45:35 INFO 05:45:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:45:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:45:35 INFO [loop_until]: OK (rc = 0) 05:45:35 DEBUG --- stdout --- 05:45:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 227m 1% 3456Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 190m 0% 18871Mi 66% 05:45:35 DEBUG --- stderr --- 05:45:35 DEBUG 05:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364761" 05:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364761" 05:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364761" 05:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364761" 05:46:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364761" 05:46:01 INFO [http_cmd]: http status code OK 05:46:01 DEBUG --- status code --- 05:46:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:01 DEBUG --- http response --- 05:46:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:01 INFO [http_cmd]: http status code OK 05:46:01 DEBUG --- status code --- 05:46:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:01 DEBUG --- http response --- 05:46:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:01 INFO [http_cmd]: http status code OK 05:46:01 INFO [http_cmd]: http status code OK 05:46:01 INFO [http_cmd]: http status code OK 05:46:01 DEBUG --- status code --- 05:46:01 DEBUG --- status code --- 05:46:01 DEBUG --- status code --- 05:46:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:01 DEBUG --- http response --- 05:46:01 DEBUG --- http response --- 05:46:01 DEBUG --- http response --- 05:46:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 ERROR [http_cmd]: ERROR 05:46:02 ERROR --- status code --- 05:46:02 DEBUG --- status code --- 05:46:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 ERROR --- http response --- 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364762" 05:46:02 DEBUG --- status code --- 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364762" 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364762" 05:46:02 DEBUG --- status code --- 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG --- http response --- 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG --- http response --- 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- http response --- 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- http response --- 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:46:02 DEBUG { 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG { 05:46:02 DEBUG { 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG { 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG { 05:46:02 DEBUG { 05:46:02 DEBUG "data": { 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG "data": { 05:46:02 DEBUG "data": { 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "data": { 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "data": { 05:46:02 DEBUG "data": { 05:46:02 DEBUG "result": [ 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "result": [ 05:46:02 DEBUG "result": [ 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG { 05:46:02 DEBUG "result": [ 05:46:02 DEBUG { 05:46:02 DEBUG { 05:46:02 DEBUG "result": [ 05:46:02 DEBUG "result": [ 05:46:02 DEBUG "metric": { 05:46:02 DEBUG { 05:46:02 DEBUG "metric": { 05:46:02 DEBUG "metric": { 05:46:02 DEBUG { 05:46:02 DEBUG { 05:46:02 DEBUG "pod": "ds-idrepo-0" 05:46:02 DEBUG "metric": { 05:46:02 DEBUG "pod": "ds-idrepo-0" 05:46:02 DEBUG "instance": "10.142.0.11:9100" 05:46:02 DEBUG "metric": { 05:46:02 DEBUG "metric": { 05:46:02 DEBUG }, 05:46:02 DEBUG "instance": "10.142.0.11:9100" 05:46:02 DEBUG }, 05:46:02 DEBUG }, 05:46:02 DEBUG "instance": "10.142.0.11:9100" 05:46:02 DEBUG "pod": "ds-idrepo-0" 05:46:02 DEBUG "value": [ 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG "value": [ 05:46:02 DEBUG }, 05:46:02 DEBUG }, 05:46:02 DEBUG 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG "value": [ 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG [...] 05:46:02 DEBUG 05:46:02 DEBUG [...] 05:46:02 DEBUG [...] 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG [...] 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG [...] 05:46:02 DEBUG [...] 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:46:02 DEBUG 05:46:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:46:02 DEBUG "instance": "10.142.0.51:9100" 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG }, 05:46:02 DEBUG "instance": "10.142.0.51:9100" 05:46:02 DEBUG }, 05:46:02 DEBUG }, 05:46:02 DEBUG "instance": "10.142.0.51:9100" 05:46:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:46:02 DEBUG "value": [ 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG "value": [ 05:46:02 DEBUG }, 05:46:02 DEBUG }, 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG "value": [ 05:46:02 DEBUG "831672320" 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG "0" 05:46:02 DEBUG "2389382.166666667" 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG ] 05:46:02 DEBUG "0" 05:46:02 DEBUG ] 05:46:02 DEBUG ] 05:46:02 DEBUG "0.00036666666666936487" 05:46:02 DEBUG "380.73991448224575" 05:46:02 DEBUG } 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG ] 05:46:02 DEBUG ] 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG ] 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG ] 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG { 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG "data": { 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "result": [ 05:46:02 DEBUG { 05:46:02 DEBUG "metric": { 05:46:02 DEBUG "instance": "10.142.0.11:9100" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG [...] 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG "instance": "10.142.0.51:9100" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG "2402803.266666667" 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364762" 05:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364762" 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG { 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG "data": { 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "result": [ 05:46:02 DEBUG { 05:46:02 DEBUG "metric": { 05:46:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG [...] 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 1735364762, 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG "14879408128" 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG ] 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG } 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG ] 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:46:02 DEBUG } 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG } 05:46:02 DEBUG { 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG "data": { 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "result": [ 05:46:02 DEBUG { 05:46:02 DEBUG "metric": { 05:46:02 DEBUG "instance": "10.142.0.11:9100" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG [...] 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG "instance": "10.142.0.51:9100" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG "25668.26666666667" 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG { 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG "data": { 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "result": [ 05:46:02 DEBUG { 05:46:02 DEBUG "metric": { 05:46:02 DEBUG "pod": "ds-idrepo-0" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG [...] 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG "0.0008824959595894928" 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:02 INFO [http_cmd]: http status code OK 05:46:02 DEBUG --- status code --- 05:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:02 DEBUG --- http response --- 05:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:46:02 DEBUG ----- output ----- 05:46:02 DEBUG { 05:46:02 DEBUG "status": "success", 05:46:02 DEBUG "data": { 05:46:02 DEBUG "resultType": "vector", 05:46:02 DEBUG "result": [ 05:46:02 DEBUG { 05:46:02 DEBUG "metric": { 05:46:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG [...] 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG 05:46:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:46:02 DEBUG }, 05:46:02 DEBUG "value": [ 05:46:02 DEBUG 1735364762, 05:46:02 DEBUG "0.18167982760291565" 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG ] 05:46:02 DEBUG } 05:46:02 DEBUG } 05:46:33 INFO 05:46:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:46:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:46:33 INFO [loop_until]: OK (rc = 0) 05:46:33 DEBUG --- stdout --- 05:46:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 114m 13377Mi rcs-687d95c5b-qs2xv 135m 427Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:46:33 DEBUG --- stderr --- 05:46:33 DEBUG 05:46:35 INFO 05:46:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:46:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:46:35 INFO [loop_until]: OK (rc = 0) 05:46:35 DEBUG --- stdout --- 05:46:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 244m 1% 3459Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 222m 0% 18871Mi 66% 05:46:35 DEBUG --- stderr --- 05:46:35 DEBUG 05:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364821" 05:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364821" 05:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364821" 05:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364821" 05:47:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364821" 05:47:01 INFO [http_cmd]: http status code OK 05:47:01 DEBUG --- status code --- 05:47:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:01 DEBUG --- http response --- 05:47:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:01 INFO [http_cmd]: http status code OK 05:47:01 INFO [http_cmd]: http status code OK 05:47:01 DEBUG --- status code --- 05:47:01 INFO [http_cmd]: http status code OK 05:47:01 INFO [http_cmd]: http status code OK 05:47:01 DEBUG --- status code --- 05:47:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:01 DEBUG --- status code --- 05:47:01 DEBUG --- status code --- 05:47:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:01 DEBUG --- http response --- 05:47:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:01 DEBUG --- http response --- 05:47:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:01 DEBUG --- http response --- 05:47:01 DEBUG --- http response --- 05:47:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 ERROR [http_cmd]: ERROR 05:47:02 ERROR --- status code --- 05:47:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:47:02 ERROR --- http response --- 05:47:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:47:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364822" 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364822" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364822" 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG --- http response --- 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG { 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG { 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG "data": { 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG "data": { 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:47:02 DEBUG "result": [ 05:47:02 DEBUG { 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG { 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG { 05:47:02 DEBUG "metric": { 05:47:02 DEBUG "instance": "10.142.0.11:9100" 05:47:02 DEBUG }, 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG { 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364822" 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG { 05:47:02 DEBUG "result": [ 05:47:02 DEBUG "value": [ 05:47:02 DEBUG "data": { 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG "data": { 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG { 05:47:02 DEBUG 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG "data": { 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG "data": { 05:47:02 DEBUG "metric": { 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG "result": [ 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG "pod": "ds-idrepo-0" 05:47:02 DEBUG }, 05:47:02 DEBUG "result": [ 05:47:02 DEBUG "result": [ 05:47:02 DEBUG { 05:47:02 DEBUG "result": [ 05:47:02 DEBUG [...] 05:47:02 DEBUG "value": [ 05:47:02 DEBUG { 05:47:02 DEBUG { 05:47:02 DEBUG "metric": { 05:47:02 DEBUG { 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG "metric": { 05:47:02 DEBUG "metric": { 05:47:02 DEBUG "pod": "ds-idrepo-0" 05:47:02 DEBUG "metric": { 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG [...] 05:47:02 DEBUG 05:47:02 DEBUG }, 05:47:02 DEBUG 05:47:02 DEBUG "instance": "10.142.0.11:9100" 05:47:02 DEBUG "pod": "ds-idrepo-0" 05:47:02 DEBUG 05:47:02 DEBUG "instance": "10.142.0.11:9100" 05:47:02 DEBUG "value": [ 05:47:02 DEBUG "instance": "10.142.0.51:9100" 05:47:02 DEBUG }, 05:47:02 DEBUG }, 05:47:02 DEBUG 05:47:02 DEBUG }, 05:47:02 DEBUG 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG "value": [ 05:47:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG }, 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 05:47:02 DEBUG [...] 05:47:02 DEBUG "0.0013333333333330683" 05:47:02 DEBUG ] 05:47:02 DEBUG 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG } 05:47:02 DEBUG 05:47:02 DEBUG [...] 05:47:02 DEBUG "0" 05:47:02 DEBUG [...] 05:47:02 DEBUG 05:47:02 DEBUG ] 05:47:02 DEBUG [...] 05:47:02 DEBUG 05:47:02 DEBUG ] 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG } 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG } 05:47:02 DEBUG 05:47:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:47:02 DEBUG } 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG ] 05:47:02 DEBUG 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG "470.15610651974293" 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG "instance": "10.142.0.51:9100" 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG "instance": "10.142.0.51:9100" 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364822" 05:47:02 DEBUG 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364822" 05:47:02 DEBUG }, 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG }, 05:47:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:47:02 DEBUG } 05:47:02 DEBUG "value": [ 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG "value": [ 05:47:02 DEBUG }, 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG "0.0012000000000028878" 05:47:02 DEBUG "1778361.366666667" 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364822" 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG ] 05:47:02 DEBUG ] 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG "831672320" 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:02 DEBUG { 05:47:02 DEBUG ] 05:47:02 DEBUG ] 05:47:02 DEBUG ] 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:02 DEBUG "data": { 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG } 05:47:02 DEBUG "result": [ 05:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364822" 05:47:02 DEBUG } 05:47:02 DEBUG { 05:47:02 DEBUG "metric": { 05:47:02 DEBUG "instance": "10.142.0.11:9100" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG [...] 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG "instance": "10.142.0.51:9100" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG "1796043.4333333336" 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG { 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG "data": { 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG "result": [ 05:47:02 DEBUG { 05:47:02 DEBUG "metric": { 05:47:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG [...] 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG "14878531584" 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG { 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG "data": { 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG "result": [ 05:47:02 DEBUG { 05:47:02 DEBUG "metric": { 05:47:02 DEBUG "instance": "10.142.0.11:9100" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG [...] 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG "instance": "10.142.0.51:9100" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG "31266.13333333334" 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG { 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG "data": { 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG "result": [ 05:47:02 DEBUG { 05:47:02 DEBUG "metric": { 05:47:02 DEBUG "pod": "ds-idrepo-0" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG [...] 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG "0.0011640442411668802" 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:02 INFO [http_cmd]: http status code OK 05:47:02 DEBUG --- status code --- 05:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:02 DEBUG --- http response --- 05:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:47:02 DEBUG ----- output ----- 05:47:02 DEBUG { 05:47:02 DEBUG "status": "success", 05:47:02 DEBUG "data": { 05:47:02 DEBUG "resultType": "vector", 05:47:02 DEBUG "result": [ 05:47:02 DEBUG { 05:47:02 DEBUG "metric": { 05:47:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG [...] 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG 05:47:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:47:02 DEBUG }, 05:47:02 DEBUG "value": [ 05:47:02 DEBUG 1735364822, 05:47:02 DEBUG "0.12136455408121419" 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG ] 05:47:02 DEBUG } 05:47:02 DEBUG } 05:47:33 INFO 05:47:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:47:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:47:33 INFO [loop_until]: OK (rc = 0) 05:47:33 DEBUG --- stdout --- 05:47:33 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 105m 13377Mi rcs-687d95c5b-qs2xv 97m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:47:33 DEBUG --- stderr --- 05:47:33 DEBUG 05:47:35 INFO 05:47:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:47:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:47:36 INFO [loop_until]: OK (rc = 0) 05:47:36 DEBUG --- stdout --- 05:47:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 189m 1% 3448Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 144m 0% 18871Mi 66% 05:47:36 DEBUG --- stderr --- 05:47:36 DEBUG 05:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364881" 05:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364881" 05:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364881" 05:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364881" 05:48:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364881" 05:48:01 INFO [http_cmd]: http status code OK 05:48:01 INFO [http_cmd]: http status code OK 05:48:01 DEBUG --- status code --- 05:48:01 DEBUG --- status code --- 05:48:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:01 DEBUG --- http response --- 05:48:01 DEBUG --- http response --- 05:48:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:01 INFO [http_cmd]: http status code OK 05:48:01 DEBUG --- status code --- 05:48:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:01 DEBUG --- http response --- 05:48:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:01 INFO [http_cmd]: http status code OK 05:48:01 INFO [http_cmd]: http status code OK 05:48:01 DEBUG --- status code --- 05:48:01 DEBUG --- status code --- 05:48:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:01 DEBUG --- http response --- 05:48:01 DEBUG --- http response --- 05:48:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 ERROR [http_cmd]: ERROR 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG --- status code --- 05:48:02 ERROR --- status code --- 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- http response --- 05:48:02 ERROR --- http response --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- status code --- 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG --- status code --- 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364882" 05:48:02 DEBUG --- http response --- 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG --- status code --- 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG --- status code --- 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364882" 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364882" 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364882" 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364882" 05:48:02 DEBUG --- http response --- 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364882" 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364882" 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364882" 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364882" 05:48:02 DEBUG --- status code --- 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:48:02 DEBUG ----- output ----- 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- status code --- 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG { 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG { 05:48:02 DEBUG { 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG { 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG "data": { 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG { 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG { 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG "data": { 05:48:02 DEBUG "data": { 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:48:02 INFO [http_cmd]: http status code OK 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG "data": { 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG "result": [ 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG --- status code --- 05:48:02 DEBUG { 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG "data": { 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG "data": { 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG "result": [ 05:48:02 DEBUG "result": [ 05:48:02 DEBUG { 05:48:02 DEBUG { 05:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG { 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG "result": [ 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG { 05:48:02 DEBUG { 05:48:02 DEBUG { 05:48:02 DEBUG "metric": { 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG --- http response --- 05:48:02 DEBUG "data": { 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG "result": [ 05:48:02 DEBUG { 05:48:02 DEBUG "result": [ 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG "metric": { 05:48:02 DEBUG "metric": { 05:48:02 DEBUG "instance": "10.142.0.11:9100" 05:48:02 DEBUG "data": { 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:48:02 DEBUG "data": { 05:48:02 DEBUG { 05:48:02 DEBUG "metric": { 05:48:02 DEBUG { 05:48:02 DEBUG "data": { 05:48:02 DEBUG "pod": "ds-idrepo-0" 05:48:02 DEBUG "pod": "ds-idrepo-0" 05:48:02 DEBUG }, 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG "result": [ 05:48:02 DEBUG ----- output ----- 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG "metric": { 05:48:02 DEBUG "instance": "10.142.0.11:9100" 05:48:02 DEBUG "metric": { 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG }, 05:48:02 DEBUG }, 05:48:02 DEBUG "value": [ 05:48:02 DEBUG "result": [ 05:48:02 DEBUG { 05:48:02 DEBUG { 05:48:02 DEBUG "result": [ 05:48:02 DEBUG "instance": "10.142.0.11:9100" 05:48:02 DEBUG }, 05:48:02 DEBUG "instance": "10.142.0.11:9100" 05:48:02 DEBUG "result": [ 05:48:02 DEBUG "value": [ 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 05:48:02 DEBUG { 05:48:02 DEBUG "metric": { 05:48:02 DEBUG "status": "success", 05:48:02 DEBUG { 05:48:02 DEBUG }, 05:48:02 DEBUG "value": [ 05:48:02 DEBUG }, 05:48:02 DEBUG { 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "metric": { 05:48:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:48:02 DEBUG "data": { 05:48:02 DEBUG "metric": { 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 05:48:02 DEBUG "value": [ 05:48:02 DEBUG "metric": { 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "instance": "10.142.0.11:9100" 05:48:02 DEBUG }, 05:48:02 DEBUG "resultType": "vector", 05:48:02 DEBUG "pod": "ds-idrepo-0" 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "pod": "ds-idrepo-0" 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG [...] 05:48:02 DEBUG }, 05:48:02 DEBUG "value": [ 05:48:02 DEBUG "result": [ 05:48:02 DEBUG }, 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG }, 05:48:02 DEBUG [...] 05:48:02 DEBUG [...] 05:48:02 DEBUG 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 05:48:02 DEBUG { 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 05:48:02 DEBUG [...] 05:48:02 DEBUG 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "metric": { 05:48:02 DEBUG 05:48:02 DEBUG [...] 05:48:02 DEBUG 05:48:02 DEBUG [...] 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "instance": "10.142.0.51:9100" 05:48:02 DEBUG 05:48:02 DEBUG [...] 05:48:02 DEBUG 05:48:02 DEBUG }, 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:48:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:48:02 DEBUG }, 05:48:02 DEBUG [...] 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 05:48:02 DEBUG "instance": "10.142.0.51:9100" 05:48:02 DEBUG 05:48:02 DEBUG [...] 05:48:02 DEBUG }, 05:48:02 DEBUG }, 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 05:48:02 DEBUG [...] 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "instance": "10.142.0.51:9100" 05:48:02 DEBUG }, 05:48:02 DEBUG "instance": "10.142.0.51:9100" 05:48:02 DEBUG 05:48:02 DEBUG "value": [ 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:48:02 DEBUG 05:48:02 DEBUG }, 05:48:02 DEBUG "value": [ 05:48:02 DEBUG }, 05:48:02 DEBUG 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG "0.00033333333333303023" 05:48:02 DEBUG 05:48:02 DEBUG 05:48:02 DEBUG }, 05:48:02 DEBUG 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG "1221040.466666667" 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG ] 05:48:02 DEBUG "instance": "10.142.0.51:9100" 05:48:02 DEBUG 05:48:02 DEBUG "value": [ 05:48:02 DEBUG [...] 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG "value": [ 05:48:02 DEBUG 05:48:02 DEBUG "0" 05:48:02 DEBUG "637.8323665667457" 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG }, 05:48:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG 05:48:02 DEBUG "0.0006666666666707975" 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:48:02 DEBUG ] 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG "value": [ 05:48:02 DEBUG }, 05:48:02 DEBUG "14878216192" 05:48:02 DEBUG 05:48:02 DEBUG ] 05:48:02 DEBUG "1239053.5666666667" 05:48:02 DEBUG }, 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG "value": [ 05:48:02 DEBUG ] 05:48:02 DEBUG 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG "value": [ 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG "27852.800000000003" 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG } 05:48:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG "0.0010686786837129855" 05:48:02 DEBUG ] 05:48:02 DEBUG }, 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG "831672320" 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG "value": [ 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG 1735364882, 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG "0.08976545316298663" 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:02 DEBUG ] 05:48:02 DEBUG } 05:48:02 DEBUG } 05:48:33 INFO 05:48:33 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:48:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:48:34 INFO [loop_until]: OK (rc = 0) 05:48:34 DEBUG --- stdout --- 05:48:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 65m 13380Mi rcs-687d95c5b-qs2xv 63m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:48:34 DEBUG --- stderr --- 05:48:34 DEBUG 05:48:36 INFO 05:48:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:48:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:48:36 INFO [loop_until]: OK (rc = 0) 05:48:36 DEBUG --- stdout --- 05:48:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 177m 1% 3446Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 134m 0% 18875Mi 66% 05:48:36 DEBUG --- stderr --- 05:48:36 DEBUG 05:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735364941" 05:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364941" 05:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364941" 05:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364941" 05:49:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364941" 05:49:01 INFO [http_cmd]: http status code OK 05:49:01 DEBUG --- status code --- 05:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:01 DEBUG --- http response --- 05:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:01 INFO [http_cmd]: http status code OK 05:49:01 DEBUG --- status code --- 05:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:01 DEBUG --- http response --- 05:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:01 INFO [http_cmd]: http status code OK 05:49:01 DEBUG --- status code --- 05:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:01 DEBUG --- http response --- 05:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:01 INFO [http_cmd]: http status code OK 05:49:01 DEBUG --- status code --- 05:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:01 DEBUG --- http response --- 05:49:01 INFO [http_cmd]: http status code OK 05:49:01 DEBUG --- status code --- 05:49:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:01 DEBUG --- http response --- 05:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 ERROR [http_cmd]: ERROR 05:49:02 ERROR --- status code --- 05:49:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:49:02 ERROR --- http response --- 05:49:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 DEBUG --- status code --- 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 DEBUG --- http response --- 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG --- status code --- 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG --- http response --- 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735364942" 05:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735364942" 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG { 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG { 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:49:02 DEBUG { 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG { 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG { 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG "data": { 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG "data": { 05:49:02 DEBUG { 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:49:02 DEBUG { 05:49:02 DEBUG "data": { 05:49:02 DEBUG { 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG "data": { 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG "data": { 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG ----- output ----- 05:49:02 INFO [http_cmd]: http status code OK 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG "result": [ 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG "result": [ 05:49:02 DEBUG "data": { 05:49:02 DEBUG { 05:49:02 DEBUG --- status code --- 05:49:02 DEBUG "data": { 05:49:02 DEBUG "result": [ 05:49:02 DEBUG "data": { 05:49:02 DEBUG { 05:49:02 DEBUG "result": [ 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:49:02 DEBUG "result": [ 05:49:02 DEBUG { 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG { 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG "metric": { 05:49:02 DEBUG { 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG { 05:49:02 DEBUG "metric": { 05:49:02 DEBUG "result": [ 05:49:02 DEBUG "data": { 05:49:02 DEBUG --- http response --- 05:49:02 DEBUG "result": [ 05:49:02 DEBUG "metric": { 05:49:02 DEBUG "result": [ 05:49:02 DEBUG "instance": "10.142.0.11:9100" 05:49:02 DEBUG "metric": { 05:49:02 DEBUG { 05:49:02 DEBUG "metric": { 05:49:02 DEBUG "instance": "10.142.0.11:9100" 05:49:02 DEBUG { 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG { 05:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:49:02 DEBUG "pod": "ds-idrepo-0" 05:49:02 DEBUG { 05:49:02 DEBUG }, 05:49:02 DEBUG "pod": "ds-idrepo-0" 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:49:02 DEBUG }, 05:49:02 DEBUG "metric": { 05:49:02 DEBUG "result": [ 05:49:02 DEBUG "metric": { 05:49:02 DEBUG ----- output ----- 05:49:02 DEBUG }, 05:49:02 DEBUG "metric": { 05:49:02 DEBUG "value": [ 05:49:02 DEBUG }, 05:49:02 DEBUG "data": { 05:49:02 DEBUG }, 05:49:02 DEBUG "value": [ 05:49:02 DEBUG "instance": "10.142.0.11:9100" 05:49:02 DEBUG { 05:49:02 DEBUG "instance": "10.142.0.11:9100" 05:49:02 DEBUG { 05:49:02 DEBUG "value": [ 05:49:02 DEBUG "instance": "10.142.0.11:9100" 05:49:02 DEBUG 05:49:02 DEBUG "value": [ 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG "value": [ 05:49:02 DEBUG 05:49:02 DEBUG }, 05:49:02 DEBUG "metric": { 05:49:02 DEBUG }, 05:49:02 DEBUG "status": "success", 05:49:02 DEBUG 05:49:02 DEBUG }, 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "result": [ 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "value": [ 05:49:02 DEBUG "pod": "ds-idrepo-0" 05:49:02 DEBUG "value": [ 05:49:02 DEBUG "data": { 05:49:02 DEBUG 05:49:02 DEBUG "value": [ 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG { 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG }, 05:49:02 DEBUG 05:49:02 DEBUG "resultType": "vector", 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG [...] 05:49:02 DEBUG 05:49:02 DEBUG "metric": { 05:49:02 DEBUG 05:49:02 DEBUG [...] 05:49:02 DEBUG 05:49:02 DEBUG "value": [ 05:49:02 DEBUG 05:49:02 DEBUG "result": [ 05:49:02 DEBUG [...] 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG [...] 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:49:02 DEBUG }, 05:49:02 DEBUG 05:49:02 DEBUG { 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "pod": "ds-idrepo-0" 05:49:02 DEBUG [...] 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "value": [ 05:49:02 DEBUG 05:49:02 DEBUG "metric": { 05:49:02 DEBUG 05:49:02 DEBUG [...] 05:49:02 DEBUG 05:49:02 DEBUG }, 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG [...] 05:49:02 DEBUG [...] 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG 05:49:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "instance": "10.142.0.51:9100" 05:49:02 DEBUG "value": [ 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "662.1004264206381" 05:49:02 DEBUG [...] 05:49:02 DEBUG }, 05:49:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:49:02 DEBUG 05:49:02 DEBUG }, 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "instance": "10.142.0.51:9100" 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG ] 05:49:02 DEBUG 05:49:02 DEBUG "value": [ 05:49:02 DEBUG }, 05:49:02 DEBUG "value": [ 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG "0" 05:49:02 DEBUG ] 05:49:02 DEBUG } 05:49:02 DEBUG ] 05:49:02 DEBUG 05:49:02 DEBUG } 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "value": [ 05:49:02 DEBUG 05:49:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:49:02 DEBUG }, 05:49:02 DEBUG 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG ] 05:49:02 DEBUG 05:49:02 DEBUG 05:49:02 DEBUG "instance": "10.142.0.51:9100" 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG 05:49:02 DEBUG }, 05:49:02 DEBUG "value": [ 05:49:02 DEBUG "instance": "10.142.0.51:9100" 05:49:02 DEBUG "instance": "10.142.0.51:9100" 05:49:02 DEBUG } 05:49:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:49:02 DEBUG 05:49:02 DEBUG }, 05:49:02 DEBUG "0.00033333333333303023" 05:49:02 DEBUG [...] 05:49:02 DEBUG "value": [ 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG }, 05:49:02 DEBUG }, 05:49:02 DEBUG } 05:49:02 DEBUG }, 05:49:02 DEBUG [...] 05:49:02 DEBUG "value": [ 05:49:02 DEBUG ] 05:49:02 DEBUG 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG "629388.1333333334" 05:49:02 DEBUG "value": [ 05:49:02 DEBUG "value": [ 05:49:02 DEBUG "value": [ 05:49:02 DEBUG 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG } 05:49:02 DEBUG 05:49:02 DEBUG "14882365440" 05:49:02 DEBUG ] 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG 05:49:02 DEBUG "646308.2" 05:49:02 DEBUG ] 05:49:02 DEBUG 05:49:02 DEBUG ] 05:49:02 DEBUG } 05:49:02 DEBUG "0.00036666666666557537" 05:49:02 DEBUG "21162.666666666668" 05:49:02 DEBUG "831672320" 05:49:02 DEBUG 05:49:02 DEBUG ] 05:49:02 DEBUG } 05:49:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:49:02 DEBUG } 05:49:02 DEBUG ] 05:49:02 DEBUG ] 05:49:02 DEBUG ] 05:49:02 DEBUG ] 05:49:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG }, 05:49:02 DEBUG ] 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG }, 05:49:02 DEBUG ] 05:49:02 DEBUG "value": [ 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG ] 05:49:02 DEBUG ] 05:49:02 DEBUG ] 05:49:02 DEBUG "value": [ 05:49:02 DEBUG } 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG 1735364942, 05:49:02 DEBUG } 05:49:02 DEBUG "0.001309844009946977" 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG "0.09636501566148722" 05:49:02 DEBUG ] 05:49:02 DEBUG ] 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG ] 05:49:02 DEBUG ] 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:02 DEBUG } 05:49:34 INFO 05:49:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:49:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:49:34 INFO [loop_until]: OK (rc = 0) 05:49:34 DEBUG --- stdout --- 05:49:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 38m 13380Mi rcs-687d95c5b-qs2xv 90m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:49:34 DEBUG --- stderr --- 05:49:34 DEBUG 05:49:36 INFO 05:49:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:49:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:49:36 INFO [loop_until]: OK (rc = 0) 05:49:36 DEBUG --- stdout --- 05:49:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 168m 1% 3450Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 148m 0% 18874Mi 66% 05:49:36 DEBUG --- stderr --- 05:49:36 DEBUG 05:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365001" 05:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365001" 05:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365001" 05:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365001" 05:50:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365001" 05:50:01 INFO [http_cmd]: http status code OK 05:50:01 DEBUG --- status code --- 05:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:01 DEBUG --- http response --- 05:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:01 INFO [http_cmd]: http status code OK 05:50:01 DEBUG --- status code --- 05:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:01 DEBUG --- http response --- 05:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:01 INFO [http_cmd]: http status code OK 05:50:01 DEBUG --- status code --- 05:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:01 DEBUG --- http response --- 05:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:01 INFO [http_cmd]: http status code OK 05:50:01 DEBUG --- status code --- 05:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:01 DEBUG --- http response --- 05:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:01 INFO [http_cmd]: http status code OK 05:50:01 DEBUG --- status code --- 05:50:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:01 DEBUG --- http response --- 05:50:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 ERROR [http_cmd]: ERROR 05:50:02 ERROR --- status code --- 05:50:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:50:02 ERROR --- http response --- 05:50:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:50:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- http response --- 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG --- status code --- 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365002" 05:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365002" 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- http response --- 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG { 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG "status": "success", 05:50:02 INFO [http_cmd]: http status code OK 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:50:02 DEBUG --- status code --- 05:50:02 DEBUG "data": { 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG "result": [ 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:50:02 DEBUG { 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { 05:50:02 DEBUG { 05:50:02 DEBUG { 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG { 05:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG { 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG { 05:50:02 DEBUG "data": { 05:50:02 DEBUG { 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG --- http response --- 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:50:02 DEBUG "metric": { 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG "data": { 05:50:02 DEBUG "data": { 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG "data": { 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:50:02 DEBUG "pod": "ds-idrepo-0" 05:50:02 DEBUG "data": { 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG "data": { 05:50:02 DEBUG "result": [ 05:50:02 DEBUG "data": { 05:50:02 DEBUG { 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG { 05:50:02 DEBUG ----- output ----- 05:50:02 DEBUG }, 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG "result": [ 05:50:02 DEBUG "result": [ 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG { 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG "result": [ 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG { 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "result": [ 05:50:02 DEBUG { 05:50:02 DEBUG { 05:50:02 DEBUG "result": [ 05:50:02 DEBUG "metric": { 05:50:02 DEBUG "result": [ 05:50:02 DEBUG "data": { 05:50:02 DEBUG { 05:50:02 DEBUG "data": { 05:50:02 DEBUG "status": "success", 05:50:02 DEBUG 05:50:02 DEBUG { 05:50:02 DEBUG "metric": { 05:50:02 DEBUG "metric": { 05:50:02 DEBUG { 05:50:02 DEBUG "instance": "10.142.0.11:9100" 05:50:02 DEBUG { 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG "metric": { 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG "data": { 05:50:02 DEBUG 05:50:02 DEBUG "metric": { 05:50:02 DEBUG "instance": "10.142.0.11:9100" 05:50:02 DEBUG "pod": "ds-idrepo-0" 05:50:02 DEBUG "metric": { 05:50:02 DEBUG }, 05:50:02 DEBUG "metric": { 05:50:02 DEBUG "result": [ 05:50:02 DEBUG "instance": "10.142.0.11:9100" 05:50:02 DEBUG "result": [ 05:50:02 DEBUG "resultType": "vector", 05:50:02 DEBUG 05:50:02 DEBUG "instance": "10.142.0.11:9100" 05:50:02 DEBUG }, 05:50:02 DEBUG }, 05:50:02 DEBUG "instance": "10.142.0.11:9100" 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:50:02 DEBUG { 05:50:02 DEBUG }, 05:50:02 DEBUG { 05:50:02 DEBUG "result": [ 05:50:02 DEBUG [...] 05:50:02 DEBUG }, 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "value": [ 05:50:02 DEBUG }, 05:50:02 DEBUG 05:50:02 DEBUG }, 05:50:02 DEBUG "metric": { 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "metric": { 05:50:02 DEBUG { 05:50:02 DEBUG 05:50:02 DEBUG "value": [ 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG "value": [ 05:50:02 DEBUG 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "pod": "ds-idrepo-0" 05:50:02 DEBUG 05:50:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:50:02 DEBUG "metric": { 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG }, 05:50:02 DEBUG 05:50:02 DEBUG }, 05:50:02 DEBUG "pod": "ds-idrepo-0" 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG [...] 05:50:02 DEBUG 05:50:02 DEBUG "value": [ 05:50:02 DEBUG 05:50:02 DEBUG "value": [ 05:50:02 DEBUG }, 05:50:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:50:02 DEBUG 05:50:02 DEBUG [...] 05:50:02 DEBUG [...] 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG [...] 05:50:02 DEBUG 05:50:02 DEBUG "value": [ 05:50:02 DEBUG }, 05:50:02 DEBUG [...] 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG [...] 05:50:02 DEBUG 05:50:02 DEBUG [...] 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG "value": [ 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:50:02 DEBUG "instance": "10.142.0.51:9100" 05:50:02 DEBUG 05:50:02 DEBUG [...] 05:50:02 DEBUG 05:50:02 DEBUG [...] 05:50:02 DEBUG 05:50:02 DEBUG "0" 05:50:02 DEBUG 05:50:02 DEBUG "instance": "10.142.0.51:9100" 05:50:02 DEBUG }, 05:50:02 DEBUG }, 05:50:02 DEBUG }, 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG 05:50:02 DEBUG [...] 05:50:02 DEBUG ] 05:50:02 DEBUG "instance": "10.142.0.51:9100" 05:50:02 DEBUG 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:50:02 DEBUG "instance": "10.142.0.51:9100" 05:50:02 DEBUG 05:50:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:50:02 DEBUG 05:50:02 DEBUG } 05:50:02 DEBUG }, 05:50:02 DEBUG 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG "1226405.9333333336" 05:50:02 DEBUG ] 05:50:02 DEBUG } 05:50:02 DEBUG }, 05:50:02 DEBUG 05:50:02 DEBUG ] 05:50:02 DEBUG "value": [ 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG "33314.13333333334" 05:50:02 DEBUG "561.2342769450001" 05:50:02 DEBUG ] 05:50:02 DEBUG } 05:50:02 DEBUG 05:50:02 DEBUG ] 05:50:02 DEBUG "value": [ 05:50:02 DEBUG 05:50:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:50:02 DEBUG }, 05:50:02 DEBUG "0.00033333333333303023" 05:50:02 DEBUG ] 05:50:02 DEBUG }, 05:50:02 DEBUG }, 05:50:02 DEBUG ] 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG } 05:50:02 DEBUG "instance": "10.142.0.51:9100" 05:50:02 DEBUG "value": [ 05:50:02 DEBUG ] 05:50:02 DEBUG } 05:50:02 DEBUG ] 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG "831672320" 05:50:02 DEBUG } 05:50:02 DEBUG }, 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG } 05:50:02 DEBUG ] 05:50:02 DEBUG } 05:50:02 DEBUG 05:50:02 DEBUG } 05:50:02 DEBUG ] 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "0.0011802476751723904" 05:50:02 DEBUG "value": [ 05:50:02 DEBUG "value": [ 05:50:02 DEBUG } 05:50:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:50:02 DEBUG } 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG ] 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG }, 05:50:02 DEBUG ] 05:50:02 DEBUG "0.0005333333333368274" 05:50:02 DEBUG } 05:50:02 DEBUG "0.11257353498455784" 05:50:02 DEBUG "1242934.366666667" 05:50:02 DEBUG "value": [ 05:50:02 DEBUG } 05:50:02 DEBUG ] 05:50:02 DEBUG ] 05:50:02 DEBUG ] 05:50:02 DEBUG ] 05:50:02 DEBUG 1735365002, 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG "14882594816" 05:50:02 DEBUG ] 05:50:02 DEBUG } 05:50:02 DEBUG ] 05:50:02 DEBUG ] 05:50:02 DEBUG ] 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG ] 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:02 DEBUG } 05:50:34 INFO 05:50:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:50:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:50:34 INFO [loop_until]: OK (rc = 0) 05:50:34 DEBUG --- stdout --- 05:50:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 75m 13380Mi rcs-687d95c5b-qs2xv 118m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:50:34 DEBUG --- stderr --- 05:50:34 DEBUG 05:50:36 INFO 05:50:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:50:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:50:36 INFO [loop_until]: OK (rc = 0) 05:50:36 DEBUG --- stdout --- 05:50:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 171m 1% 3454Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 148m 0% 18874Mi 66% 05:50:36 DEBUG --- stderr --- 05:50:36 DEBUG 05:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365061" 05:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365061" 05:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365061" 05:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365061" 05:51:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365061" 05:51:01 INFO [http_cmd]: http status code OK 05:51:01 DEBUG --- status code --- 05:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:01 DEBUG --- http response --- 05:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:01 INFO [http_cmd]: http status code OK 05:51:01 DEBUG --- status code --- 05:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:01 DEBUG --- http response --- 05:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:01 INFO [http_cmd]: http status code OK 05:51:01 DEBUG --- status code --- 05:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:01 DEBUG --- http response --- 05:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:01 INFO [http_cmd]: http status code OK 05:51:01 DEBUG --- status code --- 05:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:01 DEBUG --- http response --- 05:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:01 INFO [http_cmd]: http status code OK 05:51:01 DEBUG --- status code --- 05:51:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:01 DEBUG --- http response --- 05:51:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 ERROR [http_cmd]: ERROR 05:51:02 ERROR --- status code --- 05:51:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:51:02 ERROR --- http response --- 05:51:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:51:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365062" 05:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365062" 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG { 05:51:02 DEBUG "status": "success", 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG "data": { 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG "result": [ 05:51:02 DEBUG --- http response --- 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG --- status code --- 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG { 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:51:02 DEBUG --- status code --- 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG "metric": { 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG "instance": "10.142.0.11:9100" 05:51:02 DEBUG { 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:51:02 DEBUG --- http response --- 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:51:02 DEBUG --- http response --- 05:51:02 INFO [http_cmd]: http status code OK 05:51:02 DEBUG }, 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG --- status code --- 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "data": { 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG { 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG { 05:51:02 DEBUG { 05:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG { 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG --- http response --- 05:51:02 DEBUG { 05:51:02 DEBUG 05:51:02 DEBUG "result": [ 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG "data": { 05:51:02 DEBUG { 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG "data": { 05:51:02 DEBUG "data": { 05:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG 05:51:02 DEBUG { 05:51:02 DEBUG "data": { 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG "data": { 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG ----- output ----- 05:51:02 DEBUG "data": { 05:51:02 DEBUG [...] 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG "result": [ 05:51:02 DEBUG "data": { 05:51:02 DEBUG { 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG "result": [ 05:51:02 DEBUG "result": [ 05:51:02 DEBUG { 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG 05:51:02 DEBUG "pod": "ds-idrepo-0" 05:51:02 DEBUG "result": [ 05:51:02 DEBUG { 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG "result": [ 05:51:02 DEBUG { 05:51:02 DEBUG { 05:51:02 DEBUG "status": "success", 05:51:02 DEBUG "result": [ 05:51:02 DEBUG 05:51:02 DEBUG }, 05:51:02 DEBUG { 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "result": [ 05:51:02 DEBUG "data": { 05:51:02 DEBUG { 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "data": { 05:51:02 DEBUG { 05:51:02 DEBUG 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "instance": "10.142.0.11:9100" 05:51:02 DEBUG { 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "instance": "10.142.0.11:9100" 05:51:02 DEBUG "pod": "ds-idrepo-0" 05:51:02 DEBUG "resultType": "vector", 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "instance": "10.142.0.51:9100" 05:51:02 DEBUG 05:51:02 DEBUG "pod": "ds-idrepo-0" 05:51:02 DEBUG }, 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "result": [ 05:51:02 DEBUG "pod": "ds-idrepo-0" 05:51:02 DEBUG }, 05:51:02 DEBUG }, 05:51:02 DEBUG "result": [ 05:51:02 DEBUG "instance": "10.142.0.11:9100" 05:51:02 DEBUG }, 05:51:02 DEBUG 05:51:02 DEBUG }, 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "instance": "10.142.0.11:9100" 05:51:02 DEBUG { 05:51:02 DEBUG }, 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "value": [ 05:51:02 DEBUG { 05:51:02 DEBUG }, 05:51:02 DEBUG "value": [ 05:51:02 DEBUG 05:51:02 DEBUG "value": [ 05:51:02 DEBUG 05:51:02 DEBUG }, 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "value": [ 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG "metric": { 05:51:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG [...] 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG "value": [ 05:51:02 DEBUG }, 05:51:02 DEBUG "25258.66666666667" 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG }, 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG "value": [ 05:51:02 DEBUG ] 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG [...] 05:51:02 DEBUG 05:51:02 DEBUG "value": [ 05:51:02 DEBUG 05:51:02 DEBUG [...] 05:51:02 DEBUG [...] 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG } 05:51:02 DEBUG 05:51:02 DEBUG [...] 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG [...] 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG ] 05:51:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG [...] 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:51:02 DEBUG } 05:51:02 DEBUG }, 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG [...] 05:51:02 DEBUG 05:51:02 DEBUG }, 05:51:02 DEBUG } 05:51:02 DEBUG "value": [ 05:51:02 DEBUG 05:51:02 DEBUG "instance": "10.142.0.51:9100" 05:51:02 DEBUG 05:51:02 DEBUG [...] 05:51:02 DEBUG 05:51:02 DEBUG "instance": "10.142.0.51:9100" 05:51:02 DEBUG 05:51:02 DEBUG [...] 05:51:02 DEBUG "value": [ 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:51:02 DEBUG }, 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:51:02 DEBUG }, 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG 05:51:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "instance": "10.142.0.51:9100" 05:51:02 DEBUG 05:51:02 DEBUG }, 05:51:02 DEBUG "value": [ 05:51:02 DEBUG 05:51:02 DEBUG "instance": "10.142.0.51:9100" 05:51:02 DEBUG }, 05:51:02 DEBUG }, 05:51:02 DEBUG }, 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG }, 05:51:02 DEBUG 05:51:02 DEBUG "value": [ 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG "552.8709877709689" 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "0.0003333333333335039" 05:51:02 DEBUG "value": [ 05:51:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG "2395602.733333334" 05:51:02 DEBUG "831672320" 05:51:02 DEBUG ] 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG ] 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG }, 05:51:02 DEBUG "0.0011733710247634637" 05:51:02 DEBUG ] 05:51:02 DEBUG ] 05:51:02 DEBUG } 05:51:02 DEBUG ] 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG "0.00039999999999812045" 05:51:02 DEBUG "value": [ 05:51:02 DEBUG ] 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG "2407945.3333333344" 05:51:02 DEBUG "0" 05:51:02 DEBUG "14882254848" 05:51:02 DEBUG ] 05:51:02 DEBUG ] 05:51:02 DEBUG 1735365062, 05:51:02 DEBUG } 05:51:02 DEBUG ] 05:51:02 DEBUG ] 05:51:02 DEBUG ] 05:51:02 DEBUG ] 05:51:02 DEBUG ] 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG "0.14893539074204387" 05:51:02 DEBUG ] 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG ] 05:51:02 DEBUG ] 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG ] 05:51:02 DEBUG ] 05:51:02 DEBUG ] 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG ] 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:02 DEBUG } 05:51:34 INFO 05:51:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:51:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:51:34 INFO [loop_until]: OK (rc = 0) 05:51:34 DEBUG --- stdout --- 05:51:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 121m 13381Mi rcs-687d95c5b-qs2xv 93m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:51:34 DEBUG --- stderr --- 05:51:34 DEBUG 05:51:36 INFO 05:51:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:51:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:51:36 INFO [loop_until]: OK (rc = 0) 05:51:36 DEBUG --- stdout --- 05:51:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 223m 1% 3465Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 216m 0% 18872Mi 66% 05:51:36 DEBUG --- stderr --- 05:51:36 DEBUG 05:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365121" 05:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365121" 05:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365121" 05:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365121" 05:52:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365121" 05:52:01 INFO [http_cmd]: http status code OK 05:52:01 DEBUG --- status code --- 05:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:01 DEBUG --- http response --- 05:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:01 INFO [http_cmd]: http status code OK 05:52:01 DEBUG --- status code --- 05:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:01 DEBUG --- http response --- 05:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:01 INFO [http_cmd]: http status code OK 05:52:01 DEBUG --- status code --- 05:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:01 DEBUG --- http response --- 05:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:01 INFO [http_cmd]: http status code OK 05:52:01 DEBUG --- status code --- 05:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:01 DEBUG --- http response --- 05:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:01 INFO [http_cmd]: http status code OK 05:52:01 DEBUG --- status code --- 05:52:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:01 DEBUG --- http response --- 05:52:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 ERROR [http_cmd]: ERROR 05:52:02 ERROR --- status code --- 05:52:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:52:02 ERROR --- http response --- 05:52:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:52:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- status code --- 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- status code --- 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- http response --- 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365122" 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365122" 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG { 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG "data": { 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG "result": [ 05:52:02 DEBUG { 05:52:02 DEBUG "metric": { 05:52:02 DEBUG "instance": "10.142.0.11:9100" 05:52:02 DEBUG }, 05:52:02 DEBUG "value": [ 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG [...] 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG "instance": "10.142.0.51:9100" 05:52:02 DEBUG }, 05:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365122" 05:52:02 DEBUG "value": [ 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG --- status code --- 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG "27852.800000000003" 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG ] 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- http response --- 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG } 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG ] 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG } 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG { 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:52:02 DEBUG { 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:52:02 DEBUG } 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:52:02 INFO [http_cmd]: http status code OK 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG { 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG --- status code --- 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG "data": { 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG { 05:52:02 DEBUG { 05:52:02 DEBUG { 05:52:02 DEBUG "data": { 05:52:02 DEBUG { 05:52:02 DEBUG { 05:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:02 DEBUG { 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG "data": { 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG --- http response --- 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG "result": [ 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG "data": { 05:52:02 DEBUG "data": { 05:52:02 DEBUG "data": { 05:52:02 DEBUG "result": [ 05:52:02 DEBUG "data": { 05:52:02 DEBUG "data": { 05:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:52:02 DEBUG "data": { 05:52:02 DEBUG { 05:52:02 DEBUG "result": [ 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG { 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG ----- output ----- 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG "metric": { 05:52:02 DEBUG { 05:52:02 DEBUG "result": [ 05:52:02 DEBUG "result": [ 05:52:02 DEBUG "result": [ 05:52:02 DEBUG "metric": { 05:52:02 DEBUG "result": [ 05:52:02 DEBUG "result": [ 05:52:02 DEBUG { 05:52:02 DEBUG "result": [ 05:52:02 DEBUG "pod": "ds-idrepo-0" 05:52:02 DEBUG "metric": { 05:52:02 DEBUG { 05:52:02 DEBUG { 05:52:02 DEBUG { 05:52:02 DEBUG "instance": "10.142.0.11:9100" 05:52:02 DEBUG { 05:52:02 DEBUG { 05:52:02 DEBUG "status": "success", 05:52:02 DEBUG { 05:52:02 DEBUG }, 05:52:02 DEBUG "pod": "ds-idrepo-0" 05:52:02 DEBUG "metric": { 05:52:02 DEBUG "metric": { 05:52:02 DEBUG "metric": { 05:52:02 DEBUG }, 05:52:02 DEBUG "metric": { 05:52:02 DEBUG "metric": { 05:52:02 DEBUG "data": { 05:52:02 DEBUG "metric": { 05:52:02 DEBUG "value": [ 05:52:02 DEBUG }, 05:52:02 DEBUG "instance": "10.142.0.11:9100" 05:52:02 DEBUG "instance": "10.142.0.11:9100" 05:52:02 DEBUG "pod": "ds-idrepo-0" 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "instance": "10.142.0.11:9100" 05:52:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:52:02 DEBUG "resultType": "vector", 05:52:02 DEBUG "pod": "ds-idrepo-0" 05:52:02 DEBUG 05:52:02 DEBUG "value": [ 05:52:02 DEBUG }, 05:52:02 DEBUG }, 05:52:02 DEBUG }, 05:52:02 DEBUG 05:52:02 DEBUG }, 05:52:02 DEBUG }, 05:52:02 DEBUG "result": [ 05:52:02 DEBUG }, 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "value": [ 05:52:02 DEBUG 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "value": [ 05:52:02 DEBUG { 05:52:02 DEBUG "value": [ 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG "metric": { 05:52:02 DEBUG 05:52:02 DEBUG [...] 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG [...] 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG [...] 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG }, 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG [...] 05:52:02 DEBUG [...] 05:52:02 DEBUG [...] 05:52:02 DEBUG 05:52:02 DEBUG [...] 05:52:02 DEBUG [...] 05:52:02 DEBUG "value": [ 05:52:02 DEBUG [...] 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG "instance": "10.142.0.51:9100" 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG }, 05:52:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG }, 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG "value": [ 05:52:02 DEBUG }, 05:52:02 DEBUG "instance": "10.142.0.51:9100" 05:52:02 DEBUG "instance": "10.142.0.51:9100" 05:52:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "instance": "10.142.0.51:9100" 05:52:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:52:02 DEBUG [...] 05:52:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG "value": [ 05:52:02 DEBUG }, 05:52:02 DEBUG }, 05:52:02 DEBUG }, 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG }, 05:52:02 DEBUG }, 05:52:02 DEBUG 05:52:02 DEBUG 05:52:02 DEBUG }, 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "2379934.8000000003" 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "value": [ 05:52:02 DEBUG 05:52:02 DEBUG "831672320" 05:52:02 DEBUG "value": [ 05:52:02 DEBUG "298.60031104199066" 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG ] 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:52:02 DEBUG ] 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG ] 05:52:02 DEBUG "0.0006666666666670078" 05:52:02 DEBUG "2396925.633333334" 05:52:02 DEBUG "0.0012852166236897039" 05:52:02 DEBUG } 05:52:02 DEBUG "0.001300000000002418" 05:52:02 DEBUG "0.14992818455786638" 05:52:02 DEBUG }, 05:52:02 DEBUG } 05:52:02 DEBUG "0" 05:52:02 DEBUG } 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG "value": [ 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG 1735365122, 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG } 05:52:02 DEBUG ] 05:52:02 DEBUG ] 05:52:02 DEBUG "14882951168" 05:52:02 DEBUG } 05:52:02 DEBUG ] 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG ] 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:02 DEBUG ] 05:52:02 DEBUG } 05:52:02 DEBUG } 05:52:34 INFO 05:52:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:52:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:52:34 INFO [loop_until]: OK (rc = 0) 05:52:34 DEBUG --- stdout --- 05:52:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 189m 13380Mi rcs-687d95c5b-qs2xv 142m 426Mi rcs-687d95c5b-xxxmk 2m 395Mi 05:52:34 DEBUG --- stderr --- 05:52:34 DEBUG 05:52:36 INFO 05:52:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:52:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:52:36 INFO [loop_until]: OK (rc = 0) 05:52:36 DEBUG --- stdout --- 05:52:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 233m 1% 3446Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 196m 0% 18876Mi 66% 05:52:36 DEBUG --- stderr --- 05:52:36 DEBUG 05:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365181" 05:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365181" 05:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365181" 05:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365181" 05:53:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365181" 05:53:01 INFO [http_cmd]: http status code OK 05:53:01 DEBUG --- status code --- 05:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:01 DEBUG --- http response --- 05:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:01 INFO [http_cmd]: http status code OK 05:53:01 DEBUG --- status code --- 05:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:01 DEBUG --- http response --- 05:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:01 INFO [http_cmd]: http status code OK 05:53:01 DEBUG --- status code --- 05:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:01 DEBUG --- http response --- 05:53:01 INFO [http_cmd]: http status code OK 05:53:01 DEBUG --- status code --- 05:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:01 INFO [http_cmd]: http status code OK 05:53:01 DEBUG --- http response --- 05:53:01 DEBUG --- status code --- 05:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:01 DEBUG --- http response --- 05:53:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 ERROR [http_cmd]: ERROR 05:53:02 DEBUG --- status code --- 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 ERROR --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- status code --- 05:53:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 ERROR --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG --- status code --- 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365182" 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:53:02 DEBUG ----- output ----- 05:53:02 DEBUG { 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG "data": { 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG "result": [ 05:53:02 DEBUG { 05:53:02 DEBUG "metric": { 05:53:02 DEBUG "instance": "10.142.0.11:9100" 05:53:02 DEBUG }, 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG [...] 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG "instance": "10.142.0.51:9100" 05:53:02 DEBUG }, 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG "22801.06666666667" 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365182" 05:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365182" 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:53:02 DEBUG ----- output ----- 05:53:02 DEBUG { 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG "data": { 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG "result": [ 05:53:02 DEBUG { 05:53:02 DEBUG "metric": { 05:53:02 DEBUG "pod": "ds-idrepo-0" 05:53:02 DEBUG }, 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG [...] 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:53:02 DEBUG }, 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG "831672320" 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:53:02 DEBUG ----- output ----- 05:53:02 DEBUG { 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG "data": { 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG "result": [ 05:53:02 DEBUG { 05:53:02 DEBUG "metric": { 05:53:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:53:02 DEBUG }, 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG [...] 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:53:02 DEBUG }, 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG "0.12345563592861523" 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG ----- output ----- 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG ----- output ----- 05:53:02 INFO [http_cmd]: http status code OK 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG { 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG { 05:53:02 DEBUG --- status code --- 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG "data": { 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG ----- output ----- 05:53:02 DEBUG "data": { 05:53:02 DEBUG --- http response --- 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:53:02 DEBUG { 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG ----- output ----- 05:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:53:02 DEBUG "result": [ 05:53:02 DEBUG ----- output ----- 05:53:02 DEBUG ----- output ----- 05:53:02 DEBUG ----- output ----- 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG "result": [ 05:53:02 DEBUG { 05:53:02 DEBUG ----- output ----- 05:53:02 DEBUG { 05:53:02 DEBUG { 05:53:02 DEBUG { 05:53:02 DEBUG { 05:53:02 DEBUG "data": { 05:53:02 DEBUG { 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG { 05:53:02 DEBUG "metric": { 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG "metric": { 05:53:02 DEBUG "data": { 05:53:02 DEBUG "status": "success", 05:53:02 DEBUG "instance": "10.142.0.11:9100" 05:53:02 DEBUG "data": { 05:53:02 DEBUG "data": { 05:53:02 DEBUG "data": { 05:53:02 DEBUG "result": [ 05:53:02 DEBUG "instance": "10.142.0.11:9100" 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG "data": { 05:53:02 DEBUG }, 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG { 05:53:02 DEBUG }, 05:53:02 DEBUG "result": [ 05:53:02 DEBUG "resultType": "vector", 05:53:02 DEBUG "value": [ 05:53:02 DEBUG "result": [ 05:53:02 DEBUG "result": [ 05:53:02 DEBUG "result": [ 05:53:02 DEBUG "metric": { 05:53:02 DEBUG "value": [ 05:53:02 DEBUG { 05:53:02 DEBUG "result": [ 05:53:02 DEBUG 05:53:02 DEBUG { 05:53:02 DEBUG { 05:53:02 DEBUG { 05:53:02 DEBUG "instance": "10.142.0.11:9100" 05:53:02 DEBUG 05:53:02 DEBUG "metric": { 05:53:02 DEBUG { 05:53:02 DEBUG 05:53:02 DEBUG "metric": { 05:53:02 DEBUG "metric": { 05:53:02 DEBUG "metric": { 05:53:02 DEBUG }, 05:53:02 DEBUG 05:53:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:53:02 DEBUG "metric": { 05:53:02 DEBUG 05:53:02 DEBUG "pod": "ds-idrepo-0" 05:53:02 DEBUG "pod": "ds-idrepo-0" 05:53:02 DEBUG "pod": "ds-idrepo-0" 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 05:53:02 DEBUG }, 05:53:02 DEBUG "instance": "10.142.0.11:9100" 05:53:02 DEBUG [...] 05:53:02 DEBUG }, 05:53:02 DEBUG }, 05:53:02 DEBUG }, 05:53:02 DEBUG 05:53:02 DEBUG [...] 05:53:02 DEBUG "value": [ 05:53:02 DEBUG }, 05:53:02 DEBUG 05:53:02 DEBUG "value": [ 05:53:02 DEBUG "value": [ 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG [...] 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG "instance": "10.142.0.51:9100" 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG "instance": "10.142.0.51:9100" 05:53:02 DEBUG [...] 05:53:02 DEBUG 05:53:02 DEBUG }, 05:53:02 DEBUG [...] 05:53:02 DEBUG [...] 05:53:02 DEBUG [...] 05:53:02 DEBUG 05:53:02 DEBUG }, 05:53:02 DEBUG 05:53:02 DEBUG [...] 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG "value": [ 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG "instance": "10.142.0.51:9100" 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG "1258769.0333333334" 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG 05:53:02 DEBUG }, 05:53:02 DEBUG "1241822.6333333335" 05:53:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:53:02 DEBUG 05:53:02 DEBUG ] 05:53:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:53:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:53:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:53:02 DEBUG "value": [ 05:53:02 DEBUG ] 05:53:02 DEBUG }, 05:53:02 DEBUG "instance": "10.142.0.51:9100" 05:53:02 DEBUG } 05:53:02 DEBUG }, 05:53:02 DEBUG }, 05:53:02 DEBUG }, 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG } 05:53:02 DEBUG "value": [ 05:53:02 DEBUG }, 05:53:02 DEBUG ] 05:53:02 DEBUG "value": [ 05:53:02 DEBUG "value": [ 05:53:02 DEBUG "value": [ 05:53:02 DEBUG "0.0003333333333335039" 05:53:02 DEBUG ] 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG "value": [ 05:53:02 DEBUG } 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG "14883647488" 05:53:02 DEBUG 1735365182, 05:53:02 DEBUG } 05:53:02 DEBUG "0.0010750489102513024" 05:53:02 DEBUG "0" 05:53:02 DEBUG "317.87668309339955" 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG ] 05:53:02 DEBUG "0.0003666666666655753" 05:53:02 DEBUG ] 05:53:02 DEBUG ] 05:53:02 DEBUG ] 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG ] 05:53:02 DEBUG ] 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG ] 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:02 DEBUG } 05:53:34 INFO 05:53:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:53:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:53:34 INFO [loop_until]: OK (rc = 0) 05:53:34 DEBUG --- stdout --- 05:53:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 659m 13381Mi rcs-687d95c5b-qs2xv 14m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:53:34 DEBUG --- stderr --- 05:53:34 DEBUG 05:53:36 INFO 05:53:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:53:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:53:36 INFO [loop_until]: OK (rc = 0) 05:53:36 DEBUG --- stdout --- 05:53:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 165m 1% 3446Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 195m 0% 18878Mi 66% 05:53:36 DEBUG --- stderr --- 05:53:36 DEBUG 05:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365241" 05:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365241" 05:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365241" 05:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365241" 05:54:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365241" 05:54:01 INFO [http_cmd]: http status code OK 05:54:01 DEBUG --- status code --- 05:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:01 DEBUG --- http response --- 05:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:01 INFO [http_cmd]: http status code OK 05:54:01 DEBUG --- status code --- 05:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:01 DEBUG --- http response --- 05:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:01 INFO [http_cmd]: http status code OK 05:54:01 DEBUG --- status code --- 05:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:01 DEBUG --- http response --- 05:54:01 INFO [http_cmd]: http status code OK 05:54:01 DEBUG --- status code --- 05:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:01 DEBUG --- http response --- 05:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:01 INFO [http_cmd]: http status code OK 05:54:01 DEBUG --- status code --- 05:54:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:01 DEBUG --- http response --- 05:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 ERROR [http_cmd]: ERROR 05:54:02 ERROR --- status code --- 05:54:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:54:02 ERROR --- http response --- 05:54:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:54:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- http response --- 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG --- status code --- 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG --- http response --- 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG --- status code --- 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365242" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG { 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG "data": { 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG "result": [ 05:54:02 DEBUG { 05:54:02 DEBUG "metric": { 05:54:02 DEBUG "instance": "10.142.0.11:9100" 05:54:02 DEBUG }, 05:54:02 DEBUG "value": [ 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG [...] 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG "instance": "10.142.0.51:9100" 05:54:02 DEBUG }, 05:54:02 DEBUG "value": [ 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG "25668.26666666667" 05:54:02 DEBUG ] 05:54:02 DEBUG } 05:54:02 DEBUG ] 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365242" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:54:02 DEBUG { 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG { 05:54:02 DEBUG "data": { 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG "data": { 05:54:02 DEBUG "result": [ 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG { 05:54:02 DEBUG "result": [ 05:54:02 DEBUG "metric": { 05:54:02 DEBUG { 05:54:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:54:02 DEBUG "metric": { 05:54:02 DEBUG }, 05:54:02 DEBUG "pod": "ds-idrepo-0" 05:54:02 DEBUG "value": [ 05:54:02 DEBUG }, 05:54:02 DEBUG 05:54:02 DEBUG "value": [ 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG [...] 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG [...] 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:54:02 DEBUG 05:54:02 DEBUG }, 05:54:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:54:02 DEBUG "value": [ 05:54:02 DEBUG }, 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG "value": [ 05:54:02 DEBUG "0.041213703646842736" 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG ] 05:54:02 DEBUG "831672320" 05:54:02 DEBUG } 05:54:02 DEBUG ] 05:54:02 DEBUG ] 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 DEBUG ] 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365242" 05:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365242" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG { 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG "data": { 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG "result": [ 05:54:02 DEBUG { 05:54:02 DEBUG "metric": { 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG "instance": "10.142.0.11:9100" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG }, 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG "value": [ 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:54:02 DEBUG 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG { 05:54:02 DEBUG [...] 05:54:02 DEBUG { 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG "data": { 05:54:02 DEBUG 05:54:02 DEBUG "data": { 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG "result": [ 05:54:02 DEBUG "instance": "10.142.0.51:9100" 05:54:02 DEBUG "result": [ 05:54:02 DEBUG { 05:54:02 DEBUG }, 05:54:02 DEBUG { 05:54:02 DEBUG "metric": { 05:54:02 DEBUG "value": [ 05:54:02 DEBUG "metric": { 05:54:02 DEBUG "instance": "10.142.0.11:9100" 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG "instance": "10.142.0.11:9100" 05:54:02 DEBUG }, 05:54:02 DEBUG "0.00033333333333303023" 05:54:02 DEBUG }, 05:54:02 DEBUG "value": [ 05:54:02 DEBUG ] 05:54:02 DEBUG "value": [ 05:54:02 DEBUG 05:54:02 DEBUG } 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG ] 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG } 05:54:02 DEBUG 05:54:02 DEBUG [...] 05:54:02 DEBUG } 05:54:02 DEBUG [...] 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG "instance": "10.142.0.51:9100" 05:54:02 DEBUG "instance": "10.142.0.51:9100" 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG --- http response --- 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG }, 05:54:02 DEBUG }, 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG "value": [ 05:54:02 INFO [http_cmd]: http status code OK 05:54:02 DEBUG "value": [ 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG --- status code --- 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG { 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG "92577.36666666668" 05:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:02 DEBUG "75942.70000000003" 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:54:02 DEBUG ] 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:54:02 DEBUG --- http response --- 05:54:02 DEBUG ] 05:54:02 DEBUG "data": { 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG } 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:54:02 DEBUG } 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG { 05:54:02 DEBUG { 05:54:02 DEBUG ] 05:54:02 DEBUG { 05:54:02 DEBUG ----- output ----- 05:54:02 DEBUG ] 05:54:02 DEBUG "result": [ 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG } 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG { 05:54:02 DEBUG } 05:54:02 DEBUG { 05:54:02 DEBUG "data": { 05:54:02 DEBUG "data": { 05:54:02 DEBUG } 05:54:02 DEBUG "data": { 05:54:02 DEBUG "status": "success", 05:54:02 DEBUG } 05:54:02 DEBUG "metric": { 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG "data": { 05:54:02 DEBUG "pod": "ds-idrepo-0" 05:54:02 DEBUG }, 05:54:02 DEBUG "value": [ 05:54:02 DEBUG 05:54:02 DEBUG "resultType": "vector", 05:54:02 DEBUG "result": [ 05:54:02 DEBUG "result": [ 05:54:02 DEBUG "result": [ 05:54:02 DEBUG 05:54:02 DEBUG "result": [ 05:54:02 DEBUG { 05:54:02 DEBUG { 05:54:02 DEBUG { 05:54:02 DEBUG 05:54:02 DEBUG { 05:54:02 DEBUG "metric": { 05:54:02 DEBUG "metric": { 05:54:02 DEBUG "metric": { 05:54:02 DEBUG [...] 05:54:02 DEBUG "metric": { 05:54:02 DEBUG "pod": "ds-idrepo-0" 05:54:02 DEBUG "instance": "10.142.0.11:9100" 05:54:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:54:02 DEBUG 05:54:02 DEBUG "pod": "ds-idrepo-0" 05:54:02 DEBUG }, 05:54:02 DEBUG }, 05:54:02 DEBUG }, 05:54:02 DEBUG 05:54:02 DEBUG }, 05:54:02 DEBUG "value": [ 05:54:02 DEBUG "value": [ 05:54:02 DEBUG "value": [ 05:54:02 DEBUG 05:54:02 DEBUG "value": [ 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG }, 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG "value": [ 05:54:02 DEBUG 05:54:02 DEBUG [...] 05:54:02 DEBUG [...] 05:54:02 DEBUG [...] 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG [...] 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG "0.0013260817284417017" 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG ] 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG 05:54:02 DEBUG } 05:54:02 DEBUG 05:54:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:54:02 DEBUG "instance": "10.142.0.51:9100" 05:54:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:54:02 DEBUG ] 05:54:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:54:02 DEBUG }, 05:54:02 DEBUG }, 05:54:02 DEBUG }, 05:54:02 DEBUG } 05:54:02 DEBUG }, 05:54:02 DEBUG "value": [ 05:54:02 DEBUG "value": [ 05:54:02 DEBUG "value": [ 05:54:02 DEBUG } 05:54:02 DEBUG "value": [ 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG 1735365242, 05:54:02 DEBUG "918.3247471964355" 05:54:02 DEBUG "0.00039999999999812045" 05:54:02 DEBUG "14886797312" 05:54:02 DEBUG ] 05:54:02 DEBUG "0" 05:54:02 DEBUG ] 05:54:02 DEBUG ] 05:54:02 DEBUG } 05:54:02 DEBUG ] 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 DEBUG ] 05:54:02 DEBUG } 05:54:02 DEBUG ] 05:54:02 DEBUG ] 05:54:02 DEBUG } 05:54:02 DEBUG ] 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:02 DEBUG } 05:54:34 INFO 05:54:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:54:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:54:34 INFO [loop_until]: OK (rc = 0) 05:54:34 DEBUG --- stdout --- 05:54:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13382Mi rcs-687d95c5b-qs2xv 50m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:54:34 DEBUG --- stderr --- 05:54:34 DEBUG 05:54:36 INFO 05:54:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:54:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:54:36 INFO [loop_until]: OK (rc = 0) 05:54:36 DEBUG --- stdout --- 05:54:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 113m 0% 3455Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 95m 0% 18876Mi 66% 05:54:36 DEBUG --- stderr --- 05:54:36 DEBUG 05:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365301" 05:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365301" 05:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365301" 05:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365301" 05:55:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365301" 05:55:01 INFO [http_cmd]: http status code OK 05:55:01 INFO [http_cmd]: http status code OK 05:55:01 DEBUG --- status code --- 05:55:01 DEBUG --- status code --- 05:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:01 DEBUG --- http response --- 05:55:01 DEBUG --- http response --- 05:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:01 INFO [http_cmd]: http status code OK 05:55:01 INFO [http_cmd]: http status code OK 05:55:01 DEBUG --- status code --- 05:55:01 INFO [http_cmd]: http status code OK 05:55:01 DEBUG --- status code --- 05:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:01 DEBUG --- status code --- 05:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:01 DEBUG --- http response --- 05:55:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:01 DEBUG --- http response --- 05:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:01 DEBUG --- http response --- 05:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 ERROR [http_cmd]: ERROR 05:55:02 ERROR --- status code --- 05:55:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:55:02 ERROR --- http response --- 05:55:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:55:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 DEBUG --- status code --- 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- status code --- 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG --- status code --- 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365302" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG { 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG "data": { 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG "result": [ 05:55:02 DEBUG { 05:55:02 DEBUG "metric": { 05:55:02 DEBUG "instance": "10.142.0.11:9100" 05:55:02 DEBUG }, 05:55:02 DEBUG "value": [ 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG [...] 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG "instance": "10.142.0.51:9100" 05:55:02 DEBUG }, 05:55:02 DEBUG "value": [ 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG "21435.733333333337" 05:55:02 DEBUG ] 05:55:02 DEBUG } 05:55:02 DEBUG ] 05:55:02 DEBUG } 05:55:02 DEBUG } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:55:02 DEBUG { 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG { 05:55:02 DEBUG "data": { 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG "data": { 05:55:02 DEBUG "result": [ 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG { 05:55:02 DEBUG "result": [ 05:55:02 DEBUG "metric": { 05:55:02 DEBUG { 05:55:02 DEBUG "pod": "ds-idrepo-0" 05:55:02 DEBUG "metric": { 05:55:02 DEBUG }, 05:55:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:55:02 DEBUG "value": [ 05:55:02 DEBUG }, 05:55:02 DEBUG 05:55:02 DEBUG "value": [ 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG [...] 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG [...] 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:55:02 DEBUG 05:55:02 DEBUG }, 05:55:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:55:02 DEBUG "value": [ 05:55:02 DEBUG }, 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG "value": [ 05:55:02 DEBUG "831672320" 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG ] 05:55:02 DEBUG "0.12343062857826033" 05:55:02 DEBUG } 05:55:02 DEBUG ] 05:55:02 DEBUG ] 05:55:02 DEBUG } 05:55:02 DEBUG } 05:55:02 DEBUG ] 05:55:02 DEBUG } 05:55:02 DEBUG } 05:55:02 DEBUG } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365302" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365302" 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG { 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG "data": { 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG "result": [ 05:55:02 DEBUG { 05:55:02 DEBUG "metric": { 05:55:02 DEBUG "instance": "10.142.0.11:9100" 05:55:02 DEBUG }, 05:55:02 DEBUG "value": [ 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG [...] 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG "instance": "10.142.0.51:9100" 05:55:02 DEBUG }, 05:55:02 DEBUG "value": [ 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG "0.0009999999999995644" 05:55:02 DEBUG ] 05:55:02 DEBUG } 05:55:02 DEBUG ] 05:55:02 DEBUG } 05:55:02 DEBUG } 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365302" 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365302" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG { 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG "data": { 05:55:02 DEBUG "resultType": "vector", 05:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365302" 05:55:02 DEBUG "result": [ 05:55:02 DEBUG { 05:55:02 DEBUG "metric": { 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG "instance": "10.142.0.11:9100" 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG }, 05:55:02 DEBUG "value": [ 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:55:02 DEBUG 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG [...] 05:55:02 DEBUG { 05:55:02 DEBUG 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG 05:55:02 DEBUG "data": { 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG "instance": "10.142.0.51:9100" 05:55:02 DEBUG }, 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG "result": [ 05:55:02 DEBUG "value": [ 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG { 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:55:02 DEBUG "metric": { 05:55:02 DEBUG "1215213.0666666671" 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG "instance": "10.142.0.11:9100" 05:55:02 DEBUG ] 05:55:02 DEBUG { 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG } 05:55:02 DEBUG ] 05:55:02 DEBUG "data": { 05:55:02 DEBUG }, 05:55:02 DEBUG } 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG "value": [ 05:55:02 DEBUG } 05:55:02 DEBUG "result": [ 05:55:02 DEBUG 05:55:02 DEBUG { 05:55:02 DEBUG 05:55:02 DEBUG "metric": { 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG 05:55:02 DEBUG "pod": "ds-idrepo-0" 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG [...] 05:55:02 DEBUG }, 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG 05:55:02 DEBUG "value": [ 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG "instance": "10.142.0.51:9100" 05:55:02 DEBUG 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG ----- output ----- 05:55:02 INFO [http_cmd]: http status code OK 05:55:02 DEBUG { 05:55:02 DEBUG }, 05:55:02 DEBUG [...] 05:55:02 DEBUG { 05:55:02 DEBUG { 05:55:02 DEBUG --- status code --- 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG "value": [ 05:55:02 DEBUG 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:02 DEBUG --- http response --- 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG 05:55:02 DEBUG "data": { 05:55:02 DEBUG "data": { 05:55:02 DEBUG "data": { 05:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:55:02 DEBUG "1231914.4000000001" 05:55:02 DEBUG 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG "result": [ 05:55:02 DEBUG { 05:55:02 DEBUG "metric": { 05:55:02 DEBUG ] 05:55:02 DEBUG "result": [ 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG ----- output ----- 05:55:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:55:02 DEBUG "instance": "10.142.0.11:9100" 05:55:02 DEBUG } 05:55:02 DEBUG { 05:55:02 DEBUG "result": [ 05:55:02 DEBUG { 05:55:02 DEBUG }, 05:55:02 DEBUG }, 05:55:02 DEBUG ] 05:55:02 DEBUG "metric": { 05:55:02 DEBUG { 05:55:02 DEBUG "status": "success", 05:55:02 DEBUG "value": [ 05:55:02 DEBUG "value": [ 05:55:02 DEBUG } 05:55:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:55:02 DEBUG "metric": { 05:55:02 DEBUG "data": { 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG 05:55:02 DEBUG } 05:55:02 DEBUG }, 05:55:02 DEBUG "pod": "ds-idrepo-0" 05:55:02 DEBUG "resultType": "vector", 05:55:02 DEBUG "0.0012645002482703062" 05:55:02 DEBUG 05:55:02 DEBUG "value": [ 05:55:02 DEBUG 05:55:02 DEBUG }, 05:55:02 DEBUG ] 05:55:02 DEBUG 05:55:02 DEBUG "result": [ 05:55:02 DEBUG 05:55:02 DEBUG "value": [ 05:55:02 DEBUG } 05:55:02 DEBUG [...] 05:55:02 DEBUG { 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG ] 05:55:02 DEBUG 05:55:02 DEBUG "metric": { 05:55:02 DEBUG [...] 05:55:02 DEBUG 05:55:02 DEBUG } 05:55:02 DEBUG 05:55:02 DEBUG "pod": "ds-idrepo-0" 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG } 05:55:02 DEBUG 05:55:02 DEBUG }, 05:55:02 DEBUG 05:55:02 DEBUG [...] 05:55:02 DEBUG "instance": "10.142.0.51:9100" 05:55:02 DEBUG "value": [ 05:55:02 DEBUG 05:55:02 DEBUG 05:55:02 DEBUG }, 05:55:02 DEBUG 05:55:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:55:02 DEBUG 05:55:02 DEBUG "value": [ 05:55:02 DEBUG 05:55:02 DEBUG }, 05:55:02 DEBUG 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG 05:55:02 DEBUG "value": [ 05:55:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:55:02 DEBUG "0.0010666666666651282" 05:55:02 DEBUG [...] 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG }, 05:55:02 DEBUG ] 05:55:02 DEBUG 05:55:02 DEBUG "14889095168" 05:55:02 DEBUG "value": [ 05:55:02 DEBUG } 05:55:02 DEBUG 05:55:02 DEBUG ] 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG ] 05:55:02 DEBUG 05:55:02 DEBUG } 05:55:02 DEBUG "864.8614856612735" 05:55:02 DEBUG } 05:55:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:55:02 DEBUG ] 05:55:02 DEBUG ] 05:55:02 DEBUG } 05:55:02 DEBUG }, 05:55:02 DEBUG } 05:55:02 DEBUG } 05:55:02 DEBUG "value": [ 05:55:02 DEBUG } 05:55:02 DEBUG ] 05:55:02 DEBUG 1735365302, 05:55:02 DEBUG "0" 05:55:02 DEBUG } 05:55:02 DEBUG ] 05:55:02 DEBUG } 05:55:02 DEBUG } 05:55:02 DEBUG ] 05:55:02 DEBUG } 05:55:02 DEBUG } 05:55:34 INFO 05:55:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:55:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:55:34 INFO [loop_until]: OK (rc = 0) 05:55:34 DEBUG --- stdout --- 05:55:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 52m 13381Mi rcs-687d95c5b-qs2xv 58m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:55:34 DEBUG --- stderr --- 05:55:34 DEBUG 05:55:36 INFO 05:55:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:55:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:55:36 INFO [loop_until]: OK (rc = 0) 05:55:36 DEBUG --- stdout --- 05:55:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 174m 1% 3449Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 160m 0% 18888Mi 66% 05:55:36 DEBUG --- stderr --- 05:55:36 DEBUG 05:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365361" 05:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365361" 05:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365361" 05:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365361" 05:56:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365361" 05:56:01 INFO [http_cmd]: http status code OK 05:56:01 DEBUG --- status code --- 05:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:01 INFO [http_cmd]: http status code OK 05:56:01 DEBUG --- http response --- 05:56:01 DEBUG --- status code --- 05:56:01 INFO [http_cmd]: http status code OK 05:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:01 INFO [http_cmd]: http status code OK 05:56:01 INFO [http_cmd]: http status code OK 05:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:01 DEBUG --- http response --- 05:56:01 DEBUG --- status code --- 05:56:01 DEBUG --- status code --- 05:56:01 DEBUG --- status code --- 05:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:01 DEBUG --- http response --- 05:56:01 DEBUG --- http response --- 05:56:01 DEBUG --- http response --- 05:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 ERROR [http_cmd]: ERROR 05:56:02 ERROR --- status code --- 05:56:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:56:02 ERROR --- http response --- 05:56:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:56:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG --- status code --- 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG { 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "data": { 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG "result": [ 05:56:02 DEBUG { 05:56:02 DEBUG "metric": { 05:56:02 DEBUG "instance": "10.142.0.11:9100" 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG [...] 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG "instance": "10.142.0.51:9100" 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG "29627.733333333337" 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG { 05:56:02 DEBUG { 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "data": { 05:56:02 DEBUG "data": { 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG "result": [ 05:56:02 DEBUG "result": [ 05:56:02 DEBUG { 05:56:02 DEBUG { 05:56:02 DEBUG "metric": { 05:56:02 DEBUG "metric": { 05:56:02 DEBUG "pod": "ds-idrepo-0" 05:56:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:56:02 DEBUG }, 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG [...] 05:56:02 DEBUG [...] 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:56:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:56:02 DEBUG }, 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG "831672320" 05:56:02 DEBUG "0.14464111759799395" 05:56:02 DEBUG ] 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:02 DEBUG ] 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG { 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "data": { 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG "result": [ 05:56:02 DEBUG { 05:56:02 DEBUG "metric": { 05:56:02 DEBUG "instance": "10.142.0.11:9100" 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG [...] 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG "instance": "10.142.0.51:9100" 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG "0.0003333333333335039" 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG { 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "data": { 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG "result": [ 05:56:02 DEBUG { 05:56:02 DEBUG "metric": { 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365362" 05:56:02 DEBUG "instance": "10.142.0.11:9100" 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365362" 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG [...] 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG "instance": "10.142.0.51:9100" 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG "1781115.2333333339" 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365362" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG { 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "data": { 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG "result": [ 05:56:02 DEBUG { 05:56:02 DEBUG "metric": { 05:56:02 DEBUG "instance": "10.142.0.11:9100" 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG [...] 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG "instance": "10.142.0.51:9100" 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG "1790740.7000000004" 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG { 05:56:02 DEBUG { 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "data": { 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG "data": { 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG "result": [ 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG "result": [ 05:56:02 DEBUG { 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG { 05:56:02 DEBUG "metric": { 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:56:02 DEBUG "metric": { 05:56:02 DEBUG "instance": "10.142.0.11:9100" 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG "pod": "ds-idrepo-0" 05:56:02 DEBUG }, 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG { 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG "data": { 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG "result": [ 05:56:02 INFO [http_cmd]: http status code OK 05:56:02 DEBUG 05:56:02 DEBUG [...] 05:56:02 DEBUG { 05:56:02 DEBUG { 05:56:02 DEBUG --- status code --- 05:56:02 DEBUG [...] 05:56:02 DEBUG 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG "metric": { 05:56:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG "data": { 05:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:02 DEBUG }, 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG --- http response --- 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 05:56:02 DEBUG "instance": "10.142.0.51:9100" 05:56:02 DEBUG "result": [ 05:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:56:02 DEBUG 05:56:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:56:02 DEBUG }, 05:56:02 DEBUG { 05:56:02 DEBUG ----- output ----- 05:56:02 DEBUG 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG "metric": { 05:56:02 DEBUG { 05:56:02 DEBUG 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG "pod": "ds-idrepo-0" 05:56:02 DEBUG "status": "success", 05:56:02 DEBUG [...] 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG "0.0007000000000023952" 05:56:02 DEBUG }, 05:56:02 DEBUG "data": { 05:56:02 DEBUG 05:56:02 DEBUG "0.0013929836782556634" 05:56:02 DEBUG ] 05:56:02 DEBUG "value": [ 05:56:02 DEBUG "resultType": "vector", 05:56:02 DEBUG 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG 05:56:02 DEBUG "result": [ 05:56:02 DEBUG 05:56:02 DEBUG } 05:56:02 DEBUG ] 05:56:02 DEBUG 05:56:02 DEBUG { 05:56:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:56:02 DEBUG }, 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG "metric": { 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG 05:56:02 DEBUG "14886928384" 05:56:02 DEBUG "pod": "ds-idrepo-0" 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:02 DEBUG [...] 05:56:02 DEBUG ] 05:56:02 DEBUG }, 05:56:02 DEBUG } 05:56:02 DEBUG 05:56:02 DEBUG } 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 05:56:02 DEBUG ] 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG } 05:56:02 DEBUG 05:56:02 DEBUG 05:56:02 DEBUG } 05:56:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:56:02 DEBUG [...] 05:56:02 DEBUG }, 05:56:02 DEBUG 05:56:02 DEBUG "value": [ 05:56:02 DEBUG 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG 05:56:02 DEBUG "0" 05:56:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:56:02 DEBUG ] 05:56:02 DEBUG }, 05:56:02 DEBUG } 05:56:02 DEBUG "value": [ 05:56:02 DEBUG ] 05:56:02 DEBUG 1735365362, 05:56:02 DEBUG } 05:56:02 DEBUG "1192.4575614606433" 05:56:02 DEBUG } 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG ] 05:56:02 DEBUG } 05:56:02 DEBUG } 05:56:34 INFO 05:56:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:56:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:56:34 INFO [loop_until]: OK (rc = 0) 05:56:34 DEBUG --- stdout --- 05:56:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 87m 13381Mi rcs-687d95c5b-qs2xv 83m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:56:34 DEBUG --- stderr --- 05:56:34 DEBUG 05:56:36 INFO 05:56:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:56:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:56:36 INFO [loop_until]: OK (rc = 0) 05:56:36 DEBUG --- stdout --- 05:56:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 227m 1% 3423Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 214m 0% 18875Mi 66% 05:56:36 DEBUG --- stderr --- 05:56:36 DEBUG 05:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365421" 05:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365421" 05:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365421" 05:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365421" 05:57:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365421" 05:57:01 INFO [http_cmd]: http status code OK 05:57:01 INFO [http_cmd]: http status code OK 05:57:01 INFO [http_cmd]: http status code OK 05:57:01 DEBUG --- status code --- 05:57:01 INFO [http_cmd]: http status code OK 05:57:01 DEBUG --- status code --- 05:57:01 INFO [http_cmd]: http status code OK 05:57:01 DEBUG --- status code --- 05:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:01 DEBUG --- status code --- 05:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:01 DEBUG --- status code --- 05:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:01 DEBUG --- http response --- 05:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:01 DEBUG --- http response --- 05:57:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:01 DEBUG --- http response --- 05:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:01 DEBUG --- http response --- 05:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:01 DEBUG --- http response --- 05:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 ERROR [http_cmd]: ERROR 05:57:02 ERROR --- status code --- 05:57:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:57:02 ERROR --- http response --- 05:57:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:57:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG --- status code --- 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG { 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG "data": { 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG "result": [ 05:57:02 DEBUG { 05:57:02 DEBUG "metric": { 05:57:02 DEBUG "instance": "10.142.0.11:9100" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG [...] 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG "instance": "10.142.0.51:9100" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG "24712.533333333336" 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG { 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG "data": { 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG "result": [ 05:57:02 DEBUG { 05:57:02 DEBUG "metric": { 05:57:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG [...] 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG "0.21820932178637611" 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG } 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG { 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG "data": { 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG "result": [ 05:57:02 DEBUG { 05:57:02 DEBUG "metric": { 05:57:02 DEBUG "pod": "ds-idrepo-0" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG [...] 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG "831672320" 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG { 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG "data": { 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG "result": [ 05:57:02 DEBUG { 05:57:02 DEBUG "metric": { 05:57:02 DEBUG "instance": "10.142.0.11:9100" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG [...] 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG "instance": "10.142.0.51:9100" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG "0.0013333333333325945" 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG { 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG "data": { 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG "result": [ 05:57:02 DEBUG { 05:57:02 DEBUG "metric": { 05:57:02 DEBUG "instance": "10.142.0.11:9100" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG [...] 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG "instance": "10.142.0.51:9100" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG "2382852.666666667" 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG } 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG { 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG "data": { 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG "result": [ 05:57:02 DEBUG { 05:57:02 DEBUG "metric": { 05:57:02 DEBUG "instance": "10.142.0.11:9100" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG [...] 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG "instance": "10.142.0.51:9100" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG "2400865.2000000007" 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG } 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365422" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG { 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG --- status code --- 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG "data": { 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG "result": [ 05:57:02 DEBUG { 05:57:02 DEBUG "metric": { 05:57:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:57:02 DEBUG }, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG [...] 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:57:02 DEBUG 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:57:02 DEBUG 05:57:02 DEBUG { 05:57:02 DEBUG { 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG { 05:57:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:57:02 DEBUG "data": { 05:57:02 DEBUG "data": { 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG }, 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG "data": { 05:57:02 DEBUG "value": [ 05:57:02 DEBUG "result": [ 05:57:02 DEBUG "result": [ 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG { 05:57:02 DEBUG { 05:57:02 DEBUG "result": [ 05:57:02 DEBUG "14884118528" 05:57:02 INFO [http_cmd]: http status code OK 05:57:02 DEBUG "metric": { 05:57:02 DEBUG "metric": { 05:57:02 DEBUG { 05:57:02 DEBUG ] 05:57:02 DEBUG --- status code --- 05:57:02 DEBUG "pod": "ds-idrepo-0" 05:57:02 DEBUG "instance": "10.142.0.11:9100" 05:57:02 DEBUG "metric": { 05:57:02 DEBUG } 05:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:02 DEBUG --- http response --- 05:57:02 DEBUG }, 05:57:02 DEBUG "pod": "ds-idrepo-0" 05:57:02 DEBUG ] 05:57:02 DEBUG }, 05:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:57:02 DEBUG "value": [ 05:57:02 DEBUG }, 05:57:02 DEBUG } 05:57:02 DEBUG "value": [ 05:57:02 DEBUG ----- output ----- 05:57:02 DEBUG 05:57:02 DEBUG "value": [ 05:57:02 DEBUG } 05:57:02 DEBUG 05:57:02 DEBUG { 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG "status": "success", 05:57:02 DEBUG 05:57:02 DEBUG [...] 05:57:02 DEBUG 05:57:02 DEBUG "data": { 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG [...] 05:57:02 DEBUG "resultType": "vector", 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG "result": [ 05:57:02 DEBUG [...] 05:57:02 DEBUG 05:57:02 DEBUG 05:57:02 DEBUG { 05:57:02 DEBUG 05:57:02 DEBUG "instance": "10.142.0.51:9100" 05:57:02 DEBUG 05:57:02 DEBUG "metric": { 05:57:02 DEBUG 05:57:02 DEBUG }, 05:57:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:57:02 DEBUG "pod": "ds-idrepo-0" 05:57:02 DEBUG 05:57:02 DEBUG "value": [ 05:57:02 DEBUG }, 05:57:02 DEBUG }, 05:57:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG "value": [ 05:57:02 DEBUG "value": [ 05:57:02 DEBUG }, 05:57:02 DEBUG "0.0010333333333354252" 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG 05:57:02 DEBUG "value": [ 05:57:02 DEBUG ] 05:57:02 DEBUG "0" 05:57:02 DEBUG 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG } 05:57:02 DEBUG ] 05:57:02 DEBUG 05:57:02 DEBUG "0.0012119455876762777" 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG [...] 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG ] 05:57:02 DEBUG 05:57:02 DEBUG } 05:57:02 DEBUG } 05:57:02 DEBUG } 05:57:02 DEBUG 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG 05:57:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:57:02 DEBUG } 05:57:02 DEBUG }, 05:57:02 DEBUG } 05:57:02 DEBUG "value": [ 05:57:02 DEBUG 1735365422, 05:57:02 DEBUG "874.4092881912601" 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG ] 05:57:02 DEBUG } 05:57:02 DEBUG } 05:57:34 INFO 05:57:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:57:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:57:34 INFO [loop_until]: OK (rc = 0) 05:57:34 DEBUG --- stdout --- 05:57:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 125m 13381Mi rcs-687d95c5b-qs2xv 228m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:57:34 DEBUG --- stderr --- 05:57:34 DEBUG 05:57:36 INFO 05:57:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:57:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:57:36 INFO [loop_until]: OK (rc = 0) 05:57:36 DEBUG --- stdout --- 05:57:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 237m 1% 3436Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 198m 0% 18877Mi 66% 05:57:36 DEBUG --- stderr --- 05:57:36 DEBUG 05:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365481" 05:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365481" 05:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365481" 05:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365481" 05:58:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365481" 05:58:01 INFO [http_cmd]: http status code OK 05:58:01 INFO [http_cmd]: http status code OK 05:58:01 DEBUG --- status code --- 05:58:01 DEBUG --- status code --- 05:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:01 DEBUG --- http response --- 05:58:01 DEBUG --- http response --- 05:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:01 INFO [http_cmd]: http status code OK 05:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:01 INFO [http_cmd]: http status code OK 05:58:01 DEBUG --- status code --- 05:58:01 DEBUG --- status code --- 05:58:01 INFO [http_cmd]: http status code OK 05:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:01 DEBUG --- status code --- 05:58:01 DEBUG --- http response --- 05:58:01 DEBUG --- http response --- 05:58:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:01 DEBUG --- http response --- 05:58:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 ERROR [http_cmd]: ERROR 05:58:02 ERROR --- status code --- 05:58:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:58:02 ERROR --- http response --- 05:58:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:58:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG --- status code --- 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG --- status code --- 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- status code --- 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG --- http response --- 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG { 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "data": { 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "result": [ 05:58:02 DEBUG { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "instance": "10.142.0.11:9100" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG [...] 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG "instance": "10.142.0.51:9100" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG "21299.200000000004" 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG { 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "data": { 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "result": [ 05:58:02 DEBUG { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG [...] 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG "0.08338717168813749" 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG { 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "data": { 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "result": [ 05:58:02 DEBUG { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "pod": "ds-idrepo-0" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG [...] 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG "831672320" 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG { 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "data": { 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "result": [ 05:58:02 DEBUG { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "instance": "10.142.0.11:9100" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG [...] 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG "instance": "10.142.0.51:9100" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG "0.00033333333333303023" 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG { 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "data": { 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "result": [ 05:58:02 DEBUG { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "instance": "10.142.0.11:9100" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG [...] 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG "instance": "10.142.0.51:9100" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG "2371084.9" 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG { 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "data": { 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "result": [ 05:58:02 DEBUG { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "instance": "10.142.0.11:9100" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG [...] 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG "instance": "10.142.0.51:9100" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG "2387922.333333334" 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365482" 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG { 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "data": { 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "result": [ 05:58:02 DEBUG { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG [...] 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG "14882336768" 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG { 05:58:02 DEBUG { 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "data": { 05:58:02 DEBUG "data": { 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "result": [ 05:58:02 DEBUG "result": [ 05:58:02 DEBUG { 05:58:02 DEBUG { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "instance": "10.142.0.11:9100" 05:58:02 DEBUG "pod": "ds-idrepo-0" 05:58:02 DEBUG }, 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG [...] 05:58:02 DEBUG [...] 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG "instance": "10.142.0.51:9100" 05:58:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:58:02 DEBUG }, 05:58:02 DEBUG }, 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 INFO [http_cmd]: http status code OK 05:58:02 DEBUG "value": [ 05:58:02 DEBUG "value": [ 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG --- status code --- 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:02 DEBUG "0.0005666666666722147" 05:58:02 DEBUG "0" 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG --- http response --- 05:58:02 DEBUG ] 05:58:02 DEBUG ] 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG ----- output ----- 05:58:02 DEBUG ] 05:58:02 DEBUG ] 05:58:02 DEBUG { 05:58:02 DEBUG { 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG "status": "success", 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 DEBUG "data": { 05:58:02 DEBUG "data": { 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "resultType": "vector", 05:58:02 DEBUG "result": [ 05:58:02 DEBUG "result": [ 05:58:02 DEBUG { 05:58:02 DEBUG { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "metric": { 05:58:02 DEBUG "pod": "ds-idrepo-0" 05:58:02 DEBUG "pod": "ds-idrepo-0" 05:58:02 DEBUG }, 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG [...] 05:58:02 DEBUG [...] 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG 05:58:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:58:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:58:02 DEBUG }, 05:58:02 DEBUG }, 05:58:02 DEBUG "value": [ 05:58:02 DEBUG "value": [ 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG 1735365482, 05:58:02 DEBUG "1050.4571329607534" 05:58:02 DEBUG "0.0012286719035174796" 05:58:02 DEBUG ] 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 DEBUG ] 05:58:02 DEBUG ] 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:02 DEBUG } 05:58:34 INFO 05:58:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:58:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:58:34 INFO [loop_until]: OK (rc = 0) 05:58:34 DEBUG --- stdout --- 05:58:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 1075m 13382Mi rcs-687d95c5b-qs2xv 159m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:58:34 DEBUG --- stderr --- 05:58:34 DEBUG 05:58:36 INFO 05:58:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:58:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:58:36 INFO [loop_until]: OK (rc = 0) 05:58:36 DEBUG --- stdout --- 05:58:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 235m 1% 3452Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 729m 2% 18878Mi 66% 05:58:36 DEBUG --- stderr --- 05:58:36 DEBUG 05:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365541" 05:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365541" 05:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365541" 05:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365541" 05:59:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365541" 05:59:01 INFO [http_cmd]: http status code OK 05:59:01 DEBUG --- status code --- 05:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:01 DEBUG --- http response --- 05:59:01 INFO [http_cmd]: http status code OK 05:59:01 DEBUG --- status code --- 05:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:01 DEBUG --- http response --- 05:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:01 INFO [http_cmd]: http status code OK 05:59:01 DEBUG --- status code --- 05:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:01 DEBUG --- http response --- 05:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:01 INFO [http_cmd]: http status code OK 05:59:01 INFO [http_cmd]: http status code OK 05:59:01 DEBUG --- status code --- 05:59:01 DEBUG --- status code --- 05:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:01 DEBUG --- http response --- 05:59:01 DEBUG --- http response --- 05:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 ERROR [http_cmd]: ERROR 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 ERROR --- status code --- 05:59:02 DEBUG --- status code --- 05:59:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 ERROR --- http response --- 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:59:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 DEBUG --- status code --- 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG --- status code --- 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG { 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG "data": { 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG "result": [ 05:59:02 DEBUG { 05:59:02 DEBUG "metric": { 05:59:02 DEBUG "instance": "10.142.0.11:9100" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG [...] 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG "instance": "10.142.0.51:9100" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG "24302.933333333334" 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG { 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG "data": { 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG "result": [ 05:59:02 DEBUG { 05:59:02 DEBUG "metric": { 05:59:02 DEBUG "pod": "ds-idrepo-0" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG [...] 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG "831672320" 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG } 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG { 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG "data": { 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG "result": [ 05:59:02 DEBUG { 05:59:02 DEBUG "metric": { 05:59:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365542" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG [...] 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG "0.570316049018527" 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG { 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG "data": { 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG "result": [ 05:59:02 DEBUG { 05:59:02 DEBUG "metric": { 05:59:02 DEBUG "instance": "10.142.0.11:9100" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG [...] 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG "instance": "10.142.0.51:9100" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG "0.0006666666666665342" 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG { 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG "data": { 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG "result": [ 05:59:02 DEBUG { 05:59:02 DEBUG "metric": { 05:59:02 DEBUG "instance": "10.142.0.11:9100" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG [...] 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG "instance": "10.142.0.51:9100" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG "2090950.2666666668" 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG { 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG "data": { 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG "result": [ 05:59:02 DEBUG { 05:59:02 DEBUG "metric": { 05:59:02 DEBUG "instance": "10.142.0.11:9100" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG [...] 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG "instance": "10.142.0.51:9100" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG "2106798.5666666673" 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG { 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG "data": { 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG "result": [ 05:59:02 DEBUG { 05:59:02 DEBUG "metric": { 05:59:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG [...] 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG "14886088704" 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG } 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365542" 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG { 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG "data": { 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG "result": [ 05:59:02 DEBUG { 05:59:02 DEBUG "metric": { 05:59:02 DEBUG "pod": "ds-idrepo-0" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG [...] 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 05:59:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG }, 05:59:02 DEBUG { 05:59:02 DEBUG "value": [ 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG "data": { 05:59:02 DEBUG "0" 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG ] 05:59:02 DEBUG "result": [ 05:59:02 DEBUG } 05:59:02 DEBUG { 05:59:02 DEBUG ] 05:59:02 DEBUG "metric": { 05:59:02 DEBUG } 05:59:02 DEBUG "instance": "10.142.0.11:9100" 05:59:02 DEBUG } 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG [...] 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG "instance": "10.142.0.51:9100" 05:59:02 DEBUG }, 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG "0.0005333333333320904" 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG } 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:59:02 DEBUG ----- output ----- 05:59:02 INFO [http_cmd]: http status code OK 05:59:02 DEBUG { 05:59:02 DEBUG --- status code --- 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:02 DEBUG "data": { 05:59:02 DEBUG --- http response --- 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 05:59:02 DEBUG "result": [ 05:59:02 DEBUG ----- output ----- 05:59:02 DEBUG { 05:59:02 DEBUG { 05:59:02 DEBUG "metric": { 05:59:02 DEBUG "status": "success", 05:59:02 DEBUG "pod": "ds-idrepo-0" 05:59:02 DEBUG "data": { 05:59:02 DEBUG }, 05:59:02 DEBUG "resultType": "vector", 05:59:02 DEBUG "value": [ 05:59:02 DEBUG "result": [ 05:59:02 DEBUG 05:59:02 DEBUG { 05:59:02 DEBUG 05:59:02 DEBUG "metric": { 05:59:02 DEBUG 05:59:02 DEBUG "pod": "ds-idrepo-0" 05:59:02 DEBUG [...] 05:59:02 DEBUG }, 05:59:02 DEBUG 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG 05:59:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:59:02 DEBUG 05:59:02 DEBUG }, 05:59:02 DEBUG [...] 05:59:02 DEBUG "value": [ 05:59:02 DEBUG 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG 05:59:02 DEBUG "1319.76613606689" 05:59:02 DEBUG 05:59:02 DEBUG ] 05:59:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 05:59:02 DEBUG } 05:59:02 DEBUG }, 05:59:02 DEBUG ] 05:59:02 DEBUG "value": [ 05:59:02 DEBUG } 05:59:02 DEBUG 1735365542, 05:59:02 DEBUG } 05:59:02 DEBUG "0.0011951677655922125" 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG ] 05:59:02 DEBUG } 05:59:02 DEBUG } 05:59:34 INFO 05:59:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 05:59:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:59:34 INFO [loop_until]: OK (rc = 0) 05:59:34 DEBUG --- stdout --- 05:59:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 37m 13381Mi rcs-687d95c5b-qs2xv 86m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 05:59:34 DEBUG --- stderr --- 05:59:34 DEBUG 05:59:36 INFO 05:59:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 05:59:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:59:36 INFO [loop_until]: OK (rc = 0) 05:59:36 DEBUG --- stdout --- 05:59:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 182m 1% 3454Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 148m 0% 18878Mi 66% 05:59:36 DEBUG --- stderr --- 05:59:36 DEBUG 06:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365601" 06:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365601" 06:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365601" 06:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365601" 06:00:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365601" 06:00:01 INFO [http_cmd]: http status code OK 06:00:01 DEBUG --- status code --- 06:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:01 INFO [http_cmd]: http status code OK 06:00:01 DEBUG --- http response --- 06:00:01 DEBUG --- status code --- 06:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:01 DEBUG --- http response --- 06:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:01 INFO [http_cmd]: http status code OK 06:00:01 DEBUG --- status code --- 06:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:01 INFO [http_cmd]: http status code OK 06:00:01 INFO [http_cmd]: http status code OK 06:00:01 DEBUG --- http response --- 06:00:01 DEBUG --- status code --- 06:00:01 DEBUG --- status code --- 06:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:01 DEBUG --- http response --- 06:00:01 DEBUG --- http response --- 06:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 ERROR [http_cmd]: ERROR 06:00:02 ERROR --- status code --- 06:00:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 ERROR --- http response --- 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG { 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG "data": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "result": [ 06:00:02 DEBUG { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "instance": "10.142.0.11:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG "instance": "10.142.0.51:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG "6690.133333333334" 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG { 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:00:02 DEBUG "data": { 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG { 06:00:02 DEBUG "result": [ 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG { 06:00:02 DEBUG "data": { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "pod": "ds-idrepo-0" 06:00:02 DEBUG "result": [ 06:00:02 DEBUG }, 06:00:02 DEBUG { 06:00:02 DEBUG "value": [ 06:00:02 DEBUG "metric": { 06:00:02 DEBUG 06:00:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:00:02 DEBUG 06:00:02 DEBUG }, 06:00:02 DEBUG 06:00:02 DEBUG "value": [ 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:00:02 DEBUG 06:00:02 DEBUG }, 06:00:02 DEBUG 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:00:02 DEBUG "831672320" 06:00:02 DEBUG }, 06:00:02 DEBUG ] 06:00:02 DEBUG "value": [ 06:00:02 DEBUG } 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG ] 06:00:02 DEBUG "0.057143955421794" 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG { 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG "data": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "result": [ 06:00:02 DEBUG { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "instance": "10.142.0.11:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG "instance": "10.142.0.51:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG "0.0003333333333335039" 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG { 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG "data": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "result": [ 06:00:02 DEBUG { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "instance": "10.142.0.11:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG "instance": "10.142.0.51:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG "1222210.8" 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG { 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG "data": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "result": [ 06:00:02 DEBUG { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "instance": "10.142.0.11:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG "instance": "10.142.0.51:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG "1239686.766666667" 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG { 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG "data": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "result": [ 06:00:02 DEBUG { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365602" 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG "14885978112" 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365602" 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG { 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG "data": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "result": [ 06:00:02 DEBUG { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "pod": "ds-idrepo-0" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG "0" 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG { 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG "data": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "result": [ 06:00:02 DEBUG { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "instance": "10.142.0.11:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG "instance": "10.142.0.51:9100" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG "0.0005333333333339853" 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG { 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG "data": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "result": [ 06:00:02 DEBUG { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "pod": "ds-idrepo-0" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:00:02 DEBUG }, 06:00:02 INFO [http_cmd]: http status code OK 06:00:02 DEBUG "value": [ 06:00:02 DEBUG --- status code --- 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:02 DEBUG "781.1815832528328" 06:00:02 DEBUG --- http response --- 06:00:02 DEBUG ] 06:00:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:00:02 DEBUG } 06:00:02 DEBUG ----- output ----- 06:00:02 DEBUG ] 06:00:02 DEBUG { 06:00:02 DEBUG } 06:00:02 DEBUG "status": "success", 06:00:02 DEBUG } 06:00:02 DEBUG "data": { 06:00:02 DEBUG "resultType": "vector", 06:00:02 DEBUG "result": [ 06:00:02 DEBUG { 06:00:02 DEBUG "metric": { 06:00:02 DEBUG "pod": "ds-idrepo-0" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG [...] 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG 06:00:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:00:02 DEBUG }, 06:00:02 DEBUG "value": [ 06:00:02 DEBUG 1735365602, 06:00:02 DEBUG "0.0012287085831933877" 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG ] 06:00:02 DEBUG } 06:00:02 DEBUG } 06:00:34 INFO 06:00:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:00:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:00:34 INFO [loop_until]: OK (rc = 0) 06:00:34 DEBUG --- stdout --- 06:00:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 73m 13382Mi rcs-687d95c5b-qs2xv 139m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:00:34 DEBUG --- stderr --- 06:00:34 DEBUG 06:00:36 INFO 06:00:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:00:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:00:37 INFO [loop_until]: OK (rc = 0) 06:00:37 DEBUG --- stdout --- 06:00:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 166m 1% 3451Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 142m 0% 18881Mi 66% 06:00:37 DEBUG --- stderr --- 06:00:37 DEBUG 06:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365661" 06:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365661" 06:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365661" 06:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365661" 06:01:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365661" 06:01:01 INFO [http_cmd]: http status code OK 06:01:01 DEBUG --- status code --- 06:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:01 DEBUG --- http response --- 06:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:01 INFO [http_cmd]: http status code OK 06:01:01 DEBUG --- status code --- 06:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:01 DEBUG --- http response --- 06:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:01 INFO [http_cmd]: http status code OK 06:01:01 DEBUG --- status code --- 06:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:01 INFO [http_cmd]: http status code OK 06:01:01 DEBUG --- http response --- 06:01:01 INFO [http_cmd]: http status code OK 06:01:01 DEBUG --- status code --- 06:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:01 DEBUG --- status code --- 06:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:01 DEBUG --- http response --- 06:01:01 DEBUG --- http response --- 06:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 ERROR [http_cmd]: ERROR 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 ERROR --- status code --- 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG --- status code --- 06:01:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 ERROR --- http response --- 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 DEBUG --- http response --- 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365662" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG { 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG "data": { 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG "result": [ 06:01:02 DEBUG { 06:01:02 DEBUG "metric": { 06:01:02 DEBUG "instance": "10.142.0.11:9100" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG [...] 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG "instance": "10.142.0.51:9100" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG "300236.80000000005" 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365662" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:01:02 DEBUG ----- output ----- 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG { 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG "data": { 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:01:02 DEBUG "result": [ 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG { 06:01:02 DEBUG { 06:01:02 DEBUG "metric": { 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG "pod": "ds-idrepo-0" 06:01:02 DEBUG "data": { 06:01:02 DEBUG }, 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG "value": [ 06:01:02 DEBUG "result": [ 06:01:02 DEBUG 06:01:02 DEBUG { 06:01:02 DEBUG 06:01:02 DEBUG "metric": { 06:01:02 DEBUG 06:01:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:01:02 DEBUG [...] 06:01:02 DEBUG }, 06:01:02 DEBUG 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:01:02 DEBUG 06:01:02 DEBUG }, 06:01:02 DEBUG [...] 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG 06:01:02 DEBUG "831672320" 06:01:02 DEBUG 06:01:02 DEBUG ] 06:01:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:01:02 DEBUG } 06:01:02 DEBUG }, 06:01:02 DEBUG ] 06:01:02 DEBUG "value": [ 06:01:02 DEBUG } 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG } 06:01:02 DEBUG "0.03205373207443676" 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG { 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG "data": { 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG "result": [ 06:01:02 DEBUG { 06:01:02 DEBUG "metric": { 06:01:02 DEBUG "instance": "10.142.0.11:9100" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG [...] 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG "instance": "10.142.0.51:9100" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG "0.0013333333333330685" 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365662" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG { 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG "data": { 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG "result": [ 06:01:02 DEBUG { 06:01:02 DEBUG "metric": { 06:01:02 DEBUG "instance": "10.142.0.11:9100" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG [...] 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG "instance": "10.142.0.51:9100" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG "871106.3333333335" 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG { 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG "data": { 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG "result": [ 06:01:02 DEBUG { 06:01:02 DEBUG "metric": { 06:01:02 DEBUG "instance": "10.142.0.11:9100" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG [...] 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG "instance": "10.142.0.51:9100" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG "884118.0333333334" 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365662" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG { 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG "data": { 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG "result": [ 06:01:02 DEBUG { 06:01:02 DEBUG "metric": { 06:01:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365662" 06:01:02 DEBUG [...] 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:01:02 DEBUG }, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG "14886809600" 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG { 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:01:02 DEBUG "data": { 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG { 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG "data": { 06:01:02 DEBUG "result": [ 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG { 06:01:02 DEBUG "result": [ 06:01:02 DEBUG "metric": { 06:01:02 DEBUG { 06:01:02 DEBUG "instance": "10.142.0.11:9100" 06:01:02 DEBUG "metric": { 06:01:02 DEBUG }, 06:01:02 DEBUG "pod": "ds-idrepo-0" 06:01:02 DEBUG "value": [ 06:01:02 DEBUG }, 06:01:02 DEBUG 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG [...] 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG [...] 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG 06:01:02 DEBUG "instance": "10.142.0.51:9100" 06:01:02 INFO [http_cmd]: http status code OK 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG 06:01:02 DEBUG }, 06:01:02 DEBUG --- status code --- 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:01:02 DEBUG "value": [ 06:01:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG }, 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG --- http response --- 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:01:02 DEBUG "value": [ 06:01:02 DEBUG "0.001099999999996726" 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG ] 06:01:02 DEBUG { 06:01:02 DEBUG ----- output ----- 06:01:02 DEBUG "0" 06:01:02 DEBUG } 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG { 06:01:02 DEBUG ] 06:01:02 DEBUG ] 06:01:02 DEBUG "data": { 06:01:02 DEBUG "status": "success", 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG "data": { 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG "result": [ 06:01:02 DEBUG "resultType": "vector", 06:01:02 DEBUG } 06:01:02 DEBUG { 06:01:02 DEBUG "result": [ 06:01:02 DEBUG } 06:01:02 DEBUG "metric": { 06:01:02 DEBUG { 06:01:02 DEBUG "pod": "ds-idrepo-0" 06:01:02 DEBUG "metric": { 06:01:02 DEBUG }, 06:01:02 DEBUG "pod": "ds-idrepo-0" 06:01:02 DEBUG "value": [ 06:01:02 DEBUG }, 06:01:02 DEBUG 06:01:02 DEBUG "value": [ 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG [...] 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG [...] 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG 06:01:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:01:02 DEBUG 06:01:02 DEBUG }, 06:01:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:01:02 DEBUG "value": [ 06:01:02 DEBUG }, 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG "value": [ 06:01:02 DEBUG "637.2223825579758" 06:01:02 DEBUG 1735365662, 06:01:02 DEBUG ] 06:01:02 DEBUG "0.0013329117438386794" 06:01:02 DEBUG } 06:01:02 DEBUG ] 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:02 DEBUG ] 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:02 DEBUG } 06:01:34 INFO 06:01:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:01:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:01:34 INFO [loop_until]: OK (rc = 0) 06:01:34 DEBUG --- stdout --- 06:01:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13382Mi rcs-687d95c5b-qs2xv 39m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:01:34 DEBUG --- stderr --- 06:01:34 DEBUG 06:01:37 INFO 06:01:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:01:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:01:37 INFO [loop_until]: OK (rc = 0) 06:01:37 DEBUG --- stdout --- 06:01:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 121m 0% 3400Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 89m 0% 18878Mi 66% 06:01:37 DEBUG --- stderr --- 06:01:37 DEBUG 06:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365721" 06:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365721" 06:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365721" 06:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365721" 06:02:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365721" 06:02:01 INFO [http_cmd]: http status code OK 06:02:01 DEBUG --- status code --- 06:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:01 DEBUG --- http response --- 06:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:01 INFO [http_cmd]: http status code OK 06:02:01 DEBUG --- status code --- 06:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:01 DEBUG --- http response --- 06:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:01 INFO [http_cmd]: http status code OK 06:02:01 DEBUG --- status code --- 06:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:01 DEBUG --- http response --- 06:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:01 INFO [http_cmd]: http status code OK 06:02:01 DEBUG --- status code --- 06:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:01 DEBUG --- http response --- 06:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:01 INFO [http_cmd]: http status code OK 06:02:01 DEBUG --- status code --- 06:02:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:01 DEBUG --- http response --- 06:02:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 ERROR [http_cmd]: ERROR 06:02:02 DEBUG --- status code --- 06:02:02 ERROR --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 ERROR --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:02:02 DEBUG --- status code --- 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG --- status code --- 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- status code --- 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG { 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "data": { 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG "result": [ 06:02:02 DEBUG { 06:02:02 DEBUG "metric": { 06:02:02 DEBUG "instance": "10.142.0.11:9100" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG [...] 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG "instance": "10.142.0.51:9100" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG "27170.13333333334" 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG { 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG "status": "success", 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG "data": { 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG "result": [ 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG { 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:02:02 DEBUG "metric": { 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG "pod": "ds-idrepo-0" 06:02:02 DEBUG { 06:02:02 DEBUG { 06:02:02 DEBUG }, 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "value": [ 06:02:02 DEBUG "data": { 06:02:02 DEBUG "data": { 06:02:02 DEBUG 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG 06:02:02 DEBUG "result": [ 06:02:02 DEBUG "result": [ 06:02:02 DEBUG 06:02:02 DEBUG { 06:02:02 DEBUG { 06:02:02 DEBUG [...] 06:02:02 DEBUG "metric": { 06:02:02 DEBUG "metric": { 06:02:02 DEBUG 06:02:02 DEBUG "instance": "10.142.0.11:9100" 06:02:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:02:02 DEBUG 06:02:02 DEBUG }, 06:02:02 DEBUG }, 06:02:02 DEBUG 06:02:02 DEBUG "value": [ 06:02:02 DEBUG "value": [ 06:02:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG }, 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG [...] 06:02:02 DEBUG [...] 06:02:02 DEBUG "831672320" 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG ] 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG } 06:02:02 DEBUG 06:02:02 DEBUG "instance": "10.142.0.51:9100" 06:02:02 DEBUG ] 06:02:02 DEBUG 06:02:02 DEBUG }, 06:02:02 DEBUG } 06:02:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:02:02 DEBUG "value": [ 06:02:02 DEBUG } 06:02:02 DEBUG }, 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG "0.0006666666666670078" 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG ] 06:02:02 DEBUG "0.07947053683902855" 06:02:02 DEBUG } 06:02:02 DEBUG ] 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG { 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "data": { 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG "result": [ 06:02:02 DEBUG { 06:02:02 DEBUG "metric": { 06:02:02 DEBUG "instance": "10.142.0.11:9100" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG [...] 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG "instance": "10.142.0.51:9100" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG "76055.63333333336" 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG { 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "data": { 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG "result": [ 06:02:02 DEBUG { 06:02:02 DEBUG "metric": { 06:02:02 DEBUG "instance": "10.142.0.11:9100" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG [...] 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG "instance": "10.142.0.51:9100" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG "93423.96666666667" 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365722" 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG { 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "data": { 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365722" 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG "result": [ 06:02:02 DEBUG { 06:02:02 DEBUG "metric": { 06:02:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG [...] 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG "14887358464" 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG { 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "data": { 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG "result": [ 06:02:02 DEBUG { 06:02:02 DEBUG "metric": { 06:02:02 DEBUG "pod": "ds-idrepo-0" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG [...] 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG "0" 06:02:02 DEBUG ] 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG } 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG --- status code --- 06:02:02 INFO [http_cmd]: http status code OK 06:02:02 DEBUG ] 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- status code --- 06:02:02 DEBUG } 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:02 DEBUG } 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG --- http response --- 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:02:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG ----- output ----- 06:02:02 DEBUG { 06:02:02 DEBUG { 06:02:02 DEBUG { 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "status": "success", 06:02:02 DEBUG "data": { 06:02:02 DEBUG "data": { 06:02:02 DEBUG "data": { 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG "resultType": "vector", 06:02:02 DEBUG "result": [ 06:02:02 DEBUG "result": [ 06:02:02 DEBUG "result": [ 06:02:02 DEBUG { 06:02:02 DEBUG { 06:02:02 DEBUG { 06:02:02 DEBUG "metric": { 06:02:02 DEBUG "metric": { 06:02:02 DEBUG "metric": { 06:02:02 DEBUG "pod": "ds-idrepo-0" 06:02:02 DEBUG "pod": "ds-idrepo-0" 06:02:02 DEBUG "instance": "10.142.0.11:9100" 06:02:02 DEBUG }, 06:02:02 DEBUG }, 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG "value": [ 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG [...] 06:02:02 DEBUG [...] 06:02:02 DEBUG [...] 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG 06:02:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:02:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:02:02 DEBUG "instance": "10.142.0.51:9100" 06:02:02 DEBUG }, 06:02:02 DEBUG }, 06:02:02 DEBUG }, 06:02:02 DEBUG "value": [ 06:02:02 DEBUG "value": [ 06:02:02 DEBUG "value": [ 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG 1735365722, 06:02:02 DEBUG "951.7511439496219" 06:02:02 DEBUG "0.0009002774055599818" 06:02:02 DEBUG "0.001133333333334008" 06:02:02 DEBUG ] 06:02:02 DEBUG ] 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 DEBUG ] 06:02:02 DEBUG ] 06:02:02 DEBUG ] 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:02 DEBUG } 06:02:34 INFO 06:02:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:02:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:02:34 INFO [loop_until]: OK (rc = 0) 06:02:34 DEBUG --- stdout --- 06:02:34 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13383Mi rcs-687d95c5b-qs2xv 1m 426Mi rcs-687d95c5b-xxxmk 2m 395Mi 06:02:34 DEBUG --- stderr --- 06:02:34 DEBUG 06:02:37 INFO 06:02:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:02:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:02:37 INFO [loop_until]: OK (rc = 0) 06:02:37 DEBUG --- stdout --- 06:02:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 117m 0% 3424Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 90m 0% 18878Mi 66% 06:02:37 DEBUG --- stderr --- 06:02:37 DEBUG 06:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365781" 06:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365781" 06:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365781" 06:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365781" 06:03:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365781" 06:03:01 INFO [http_cmd]: http status code OK 06:03:01 DEBUG --- status code --- 06:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:01 DEBUG --- http response --- 06:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:01 INFO [http_cmd]: http status code OK 06:03:01 DEBUG --- status code --- 06:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:01 DEBUG --- http response --- 06:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:01 INFO [http_cmd]: http status code OK 06:03:01 DEBUG --- status code --- 06:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:01 DEBUG --- http response --- 06:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:01 INFO [http_cmd]: http status code OK 06:03:01 DEBUG --- status code --- 06:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:01 DEBUG --- http response --- 06:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:01 INFO [http_cmd]: http status code OK 06:03:01 DEBUG --- status code --- 06:03:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:01 DEBUG --- http response --- 06:03:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 ERROR [http_cmd]: ERROR 06:03:02 ERROR --- status code --- 06:03:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:03:02 ERROR --- http response --- 06:03:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:03:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG --- status code --- 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG --- http response --- 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 DEBUG --- http response --- 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365782" 06:03:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG { 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "data": { 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "result": [ 06:03:02 DEBUG { 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "instance": "10.142.0.11:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG [...] 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG "instance": "10.142.0.51:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG "25668.26666666667" 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG { 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG "data": { 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG { 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG { 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "result": [ 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "data": { 06:03:02 DEBUG { 06:03:02 DEBUG "data": { 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "result": [ 06:03:02 DEBUG "pod": "ds-idrepo-0" 06:03:02 DEBUG "result": [ 06:03:02 DEBUG { 06:03:02 DEBUG }, 06:03:02 DEBUG { 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "value": [ 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:03:02 DEBUG 06:03:02 DEBUG "instance": "10.142.0.11:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 06:03:02 DEBUG [...] 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG [...] 06:03:02 DEBUG 06:03:02 DEBUG [...] 06:03:02 DEBUG 06:03:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG }, 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 06:03:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG "instance": "10.142.0.51:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG "831672320" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG ] 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG } 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG "0.0364215119235714" 06:03:02 DEBUG ] 06:03:02 DEBUG "0.00033333333333303023" 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG ] 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG { 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "data": { 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "result": [ 06:03:02 DEBUG { 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "instance": "10.142.0.11:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG [...] 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG "instance": "10.142.0.51:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG "75793.20000000003" 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG { 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "data": { 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "result": [ 06:03:02 DEBUG { 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "instance": "10.142.0.11:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG [...] 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG "instance": "10.142.0.51:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG "92265.03333333337" 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG { 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "data": { 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "result": [ 06:03:02 DEBUG { 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG [...] 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG "14886711296" 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365782" 06:03:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365782" 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG { 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "data": { 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "result": [ 06:03:02 DEBUG { 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "pod": "ds-cts-0" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG [...] 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG "pod": "ds-idrepo-0" 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG "2283123.36061102" 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 INFO [http_cmd]: http status code OK 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG --- status code --- 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG --- http response --- 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:03:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG ----- output ----- 06:03:02 DEBUG { 06:03:02 DEBUG { 06:03:02 DEBUG { 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "status": "success", 06:03:02 DEBUG "data": { 06:03:02 DEBUG "data": { 06:03:02 DEBUG "data": { 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "resultType": "vector", 06:03:02 DEBUG "result": [ 06:03:02 DEBUG "result": [ 06:03:02 DEBUG "result": [ 06:03:02 DEBUG { 06:03:02 DEBUG { 06:03:02 DEBUG { 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "metric": { 06:03:02 DEBUG "pod": "ds-cts-0" 06:03:02 DEBUG "pod": "ds-cts-0" 06:03:02 DEBUG "instance": "10.142.0.11:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG }, 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG "value": [ 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG [...] 06:03:02 DEBUG [...] 06:03:02 DEBUG [...] 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG 06:03:02 DEBUG "pod": "ds-idrepo-0" 06:03:02 DEBUG "pod": "ds-idrepo-0" 06:03:02 DEBUG "instance": "10.142.0.51:9100" 06:03:02 DEBUG }, 06:03:02 DEBUG }, 06:03:02 DEBUG }, 06:03:02 DEBUG "value": [ 06:03:02 DEBUG "value": [ 06:03:02 DEBUG "value": [ 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG 1735365782, 06:03:02 DEBUG "0.08634164349025995" 06:03:02 DEBUG "873.6441430909515" 06:03:02 DEBUG "0.00043333333333445514" 06:03:02 DEBUG ] 06:03:02 DEBUG ] 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG ] 06:03:02 DEBUG ] 06:03:02 DEBUG ] 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:02 DEBUG } 06:03:34 INFO 06:03:34 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:03:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:03:35 INFO [loop_until]: OK (rc = 0) 06:03:35 DEBUG --- stdout --- 06:03:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 12m 13382Mi rcs-687d95c5b-qs2xv 36m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:03:35 DEBUG --- stderr --- 06:03:35 DEBUG 06:03:37 INFO 06:03:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:03:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:03:37 INFO [loop_until]: OK (rc = 0) 06:03:37 DEBUG --- stdout --- 06:03:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 122m 0% 3449Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 95m 0% 18876Mi 66% 06:03:37 DEBUG --- stderr --- 06:03:37 DEBUG 06:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365841" 06:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365841" 06:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365841" 06:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365841" 06:04:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365841" 06:04:01 INFO [http_cmd]: http status code OK 06:04:01 DEBUG --- status code --- 06:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:01 DEBUG --- http response --- 06:04:01 INFO [http_cmd]: http status code OK 06:04:01 DEBUG --- status code --- 06:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:01 DEBUG --- http response --- 06:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:01 INFO [http_cmd]: http status code OK 06:04:01 DEBUG --- status code --- 06:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:01 DEBUG --- http response --- 06:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:01 INFO [http_cmd]: http status code OK 06:04:01 DEBUG --- status code --- 06:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:01 DEBUG --- http response --- 06:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:01 INFO [http_cmd]: http status code OK 06:04:01 DEBUG --- status code --- 06:04:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:01 DEBUG --- http response --- 06:04:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 ERROR [http_cmd]: ERROR 06:04:02 DEBUG --- http response --- 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 ERROR --- status code --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG --- status code --- 06:04:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 ERROR --- http response --- 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:04:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365842" 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG --- status code --- 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG { 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG "data": { 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG "result": [ 06:04:02 DEBUG { 06:04:02 DEBUG "metric": { 06:04:02 DEBUG "instance": "10.142.0.11:9100" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG [...] 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "instance": "10.142.0.51:9100" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG "25395.200000000004" 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365842" 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG { 06:04:02 DEBUG { 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG "data": { 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG "data": { 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG "result": [ 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG { 06:04:02 DEBUG { 06:04:02 DEBUG "metric": { 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG "pod": "ds-idrepo-0" 06:04:02 DEBUG "result": [ 06:04:02 DEBUG "data": { 06:04:02 DEBUG }, 06:04:02 DEBUG { 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG "value": [ 06:04:02 DEBUG "metric": { 06:04:02 DEBUG "result": [ 06:04:02 DEBUG 06:04:02 DEBUG "instance": "10.142.0.11:9100" 06:04:02 DEBUG { 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "metric": { 06:04:02 DEBUG }, 06:04:02 DEBUG [...] 06:04:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG }, 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:04:02 DEBUG 06:04:02 DEBUG [...] 06:04:02 DEBUG }, 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "value": [ 06:04:02 DEBUG [...] 06:04:02 DEBUG 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "831672320" 06:04:02 DEBUG 06:04:02 DEBUG "instance": "10.142.0.51:9100" 06:04:02 DEBUG ] 06:04:02 DEBUG 06:04:02 DEBUG }, 06:04:02 DEBUG } 06:04:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:04:02 DEBUG "value": [ 06:04:02 DEBUG ] 06:04:02 DEBUG }, 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG } 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG } 06:04:02 DEBUG "0" 06:04:02 DEBUG "0.29927765556212843" 06:04:02 DEBUG ] 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG { 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG "data": { 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG "result": [ 06:04:02 DEBUG { 06:04:02 DEBUG "metric": { 06:04:02 DEBUG "instance": "10.142.0.11:9100" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG [...] 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "instance": "10.142.0.51:9100" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG "198797.90000000005" 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365842" 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG { 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG "data": { 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG "result": [ 06:04:02 DEBUG { 06:04:02 DEBUG "metric": { 06:04:02 DEBUG "instance": "10.142.0.11:9100" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG [...] 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "instance": "10.142.0.51:9100" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG "215655.10000000012" 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG { 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG "data": { 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG "result": [ 06:04:02 DEBUG { 06:04:02 DEBUG "metric": { 06:04:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG [...] 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG "14885195776" 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365842" 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365842" 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG { 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG "data": { 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG "result": [ 06:04:02 DEBUG { 06:04:02 DEBUG "metric": { 06:04:02 DEBUG "pod": "ds-idrepo-0" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG [...] 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:04:02 DEBUG }, 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG "0" 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 INFO [http_cmd]: http status code OK 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:04:02 DEBUG --- status code --- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG { 06:04:02 DEBUG --- http response --- 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG "data": { 06:04:02 DEBUG ----- output ----- 06:04:02 DEBUG { 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG { 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG "result": [ 06:04:02 DEBUG "status": "success", 06:04:02 DEBUG "data": { 06:04:02 DEBUG { 06:04:02 DEBUG "data": { 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG "metric": { 06:04:02 DEBUG "resultType": "vector", 06:04:02 DEBUG "result": [ 06:04:02 DEBUG "instance": "10.142.0.11:9100" 06:04:02 DEBUG "result": [ 06:04:02 DEBUG { 06:04:02 DEBUG }, 06:04:02 DEBUG { 06:04:02 DEBUG "metric": { 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG "pod": "ds-idrepo-0" 06:04:02 DEBUG "metric": { 06:04:02 DEBUG 06:04:02 DEBUG }, 06:04:02 DEBUG "pod": "ds-idrepo-0" 06:04:02 DEBUG 06:04:02 DEBUG "value": [ 06:04:02 DEBUG }, 06:04:02 DEBUG [...] 06:04:02 DEBUG 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG [...] 06:04:02 DEBUG 06:04:02 DEBUG "instance": "10.142.0.51:9100" 06:04:02 DEBUG 06:04:02 DEBUG [...] 06:04:02 DEBUG }, 06:04:02 DEBUG 06:04:02 DEBUG 06:04:02 DEBUG "value": [ 06:04:02 DEBUG 06:04:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG 06:04:02 DEBUG }, 06:04:02 DEBUG "0.0005666666666684252" 06:04:02 DEBUG 06:04:02 DEBUG "value": [ 06:04:02 DEBUG ] 06:04:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG } 06:04:02 DEBUG }, 06:04:02 DEBUG "1000.6710117625038" 06:04:02 DEBUG ] 06:04:02 DEBUG "value": [ 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG 1735365842, 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 DEBUG "0.0012182142519508495" 06:04:02 DEBUG ] 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:02 DEBUG ] 06:04:02 DEBUG } 06:04:02 DEBUG } 06:04:35 INFO 06:04:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:04:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:04:35 INFO [loop_until]: OK (rc = 0) 06:04:35 DEBUG --- stdout --- 06:04:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 69m 13383Mi rcs-687d95c5b-qs2xv 61m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:04:35 DEBUG --- stderr --- 06:04:35 DEBUG 06:04:37 INFO 06:04:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:04:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:04:37 INFO [loop_until]: OK (rc = 0) 06:04:37 DEBUG --- stdout --- 06:04:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 168m 1% 3440Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 138m 0% 18879Mi 66% 06:04:37 DEBUG --- stderr --- 06:04:37 DEBUG 06:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365901" 06:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365901" 06:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365901" 06:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365901" 06:05:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365901" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365902" 06:05:02 ERROR [http_cmd]: ERROR 06:05:02 ERROR --- status code --- 06:05:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:05:02 ERROR --- http response --- 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:05:02 DEBUG --- status code --- 06:05:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG --- status code --- 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- status code --- 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- status code --- 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:02 DEBUG { 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG "data": { 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG "result": [ 06:05:02 DEBUG { 06:05:02 DEBUG "metric": { 06:05:02 DEBUG "instance": "10.142.0.11:9100" 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG [...] 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG "instance": "10.142.0.51:9100" 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG "20206.933333333338" 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365902" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG { 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG "data": { 06:05:02 DEBUG { 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG "result": [ 06:05:02 DEBUG "data": { 06:05:02 DEBUG { 06:05:02 DEBUG "metric": { 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG "instance": "10.142.0.11:9100" 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG "result": [ 06:05:02 DEBUG }, 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG { 06:05:02 DEBUG "value": [ 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG "metric": { 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG "pod": "ds-idrepo-0" 06:05:02 DEBUG }, 06:05:02 DEBUG 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:05:02 DEBUG "value": [ 06:05:02 DEBUG [...] 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG { 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG "data": { 06:05:02 DEBUG [...] 06:05:02 DEBUG "instance": "10.142.0.51:9100" 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG 06:05:02 DEBUG }, 06:05:02 DEBUG "result": [ 06:05:02 DEBUG 06:05:02 DEBUG "value": [ 06:05:02 DEBUG { 06:05:02 DEBUG 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG "metric": { 06:05:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:05:02 DEBUG "0" 06:05:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:05:02 DEBUG }, 06:05:02 DEBUG ] 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG } 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG ] 06:05:02 DEBUG 06:05:02 DEBUG "831672320" 06:05:02 DEBUG } 06:05:02 DEBUG 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG } 06:05:02 DEBUG [...] 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG ] 06:05:02 DEBUG 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG } 06:05:02 DEBUG 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG } 06:05:02 DEBUG 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:05:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG }, 06:05:02 DEBUG { 06:05:02 DEBUG "value": [ 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG "data": { 06:05:02 DEBUG "0.09936421643614919" 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG ] 06:05:02 DEBUG "result": [ 06:05:02 DEBUG } 06:05:02 DEBUG { 06:05:02 DEBUG ] 06:05:02 DEBUG "metric": { 06:05:02 DEBUG } 06:05:02 DEBUG "instance": "10.142.0.11:9100" 06:05:02 DEBUG } 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG [...] 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG "instance": "10.142.0.51:9100" 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG "1111518.1333333335" 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365902" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG { 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG "data": { 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG "result": [ 06:05:02 DEBUG { 06:05:02 DEBUG "metric": { 06:05:02 DEBUG "instance": "10.142.0.11:9100" 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG [...] 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG "instance": "10.142.0.51:9100" 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG "1128234.3000000003" 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365902" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG { 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG "data": { 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG "result": [ 06:05:02 DEBUG { 06:05:02 DEBUG "metric": { 06:05:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG [...] 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG "14886711296" 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG { 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG "data": { 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG "result": [ 06:05:02 DEBUG { 06:05:02 DEBUG "metric": { 06:05:02 DEBUG "pod": "ds-idrepo-0" 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG [...] 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:05:02 DEBUG }, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG "0" 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG } 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365902" 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG --- http response --- 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 INFO [http_cmd]: http status code OK 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG --- status code --- 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:05:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:02 DEBUG { 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG --- http response --- 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG { 06:05:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:05:02 DEBUG "data": { 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG ----- output ----- 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG "data": { 06:05:02 DEBUG { 06:05:02 DEBUG "result": [ 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG "status": "success", 06:05:02 DEBUG { 06:05:02 DEBUG "result": [ 06:05:02 DEBUG "data": { 06:05:02 DEBUG "metric": { 06:05:02 DEBUG { 06:05:02 DEBUG "resultType": "vector", 06:05:02 DEBUG "pod": "ds-idrepo-0" 06:05:02 DEBUG "metric": { 06:05:02 DEBUG "result": [ 06:05:02 DEBUG }, 06:05:02 DEBUG "pod": "ds-idrepo-0" 06:05:02 DEBUG { 06:05:02 DEBUG "value": [ 06:05:02 DEBUG }, 06:05:02 DEBUG "metric": { 06:05:02 DEBUG 06:05:02 DEBUG "value": [ 06:05:02 DEBUG "instance": "10.142.0.11:9100" 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG }, 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG "value": [ 06:05:02 DEBUG [...] 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG [...] 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG [...] 06:05:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:05:02 DEBUG 06:05:02 DEBUG 06:05:02 DEBUG }, 06:05:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:05:02 DEBUG 06:05:02 DEBUG "value": [ 06:05:02 DEBUG }, 06:05:02 DEBUG 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG "value": [ 06:05:02 DEBUG "instance": "10.142.0.51:9100" 06:05:02 DEBUG "0.0010377588869483504" 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG }, 06:05:02 DEBUG ] 06:05:02 DEBUG "419.22827607382897" 06:05:02 DEBUG "value": [ 06:05:02 DEBUG } 06:05:02 DEBUG ] 06:05:02 DEBUG 1735365902, 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG "0.0003666666666731545" 06:05:02 DEBUG } 06:05:02 DEBUG ] 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG } 06:05:02 DEBUG } 06:05:02 DEBUG } 06:05:02 DEBUG ] 06:05:02 DEBUG } 06:05:02 DEBUG } 06:05:35 INFO 06:05:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:05:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:05:35 INFO [loop_until]: OK (rc = 0) 06:05:35 DEBUG --- stdout --- 06:05:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 123m 13383Mi rcs-687d95c5b-qs2xv 1m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:05:35 DEBUG --- stderr --- 06:05:35 DEBUG 06:05:37 INFO 06:05:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:05:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:05:37 INFO [loop_until]: OK (rc = 0) 06:05:37 DEBUG --- stdout --- 06:05:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 202m 1% 3443Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 187m 0% 18879Mi 66% 06:05:37 DEBUG --- stderr --- 06:05:37 DEBUG 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 ERROR [http_cmd]: ERROR 06:06:02 DEBUG --- status code --- 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 ERROR --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 ERROR --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365962" 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG --- http response --- 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:02 DEBUG { 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG "data": { 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG "result": [ 06:06:02 DEBUG { 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "instance": "10.142.0.11:9100" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG [...] 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "instance": "10.142.0.51:9100" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG "32904.53333333334" 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365962" 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG { 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG "data": { 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG "result": [ 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG { 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG "metric": { 06:06:02 DEBUG { 06:06:02 DEBUG { 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG "instance": "10.142.0.11:9100" 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG }, 06:06:02 DEBUG "data": { 06:06:02 DEBUG "data": { 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:06:02 DEBUG "value": [ 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG 06:06:02 DEBUG "result": [ 06:06:02 DEBUG "result": [ 06:06:02 DEBUG { 06:06:02 DEBUG 06:06:02 DEBUG { 06:06:02 DEBUG { 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "data": { 06:06:02 DEBUG [...] 06:06:02 DEBUG "pod": "ds-idrepo-0" 06:06:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG 06:06:02 DEBUG }, 06:06:02 DEBUG }, 06:06:02 DEBUG "result": [ 06:06:02 DEBUG 06:06:02 DEBUG "value": [ 06:06:02 DEBUG "value": [ 06:06:02 DEBUG { 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "instance": "10.142.0.51:9100" 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "instance": "10.142.0.11:9100" 06:06:02 DEBUG }, 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG }, 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365962" 06:06:02 DEBUG "value": [ 06:06:02 DEBUG [...] 06:06:02 DEBUG [...] 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "0.00033333333333303023" 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG ] 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG } 06:06:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:06:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:06:02 DEBUG [...] 06:06:02 DEBUG ] 06:06:02 DEBUG }, 06:06:02 DEBUG }, 06:06:02 DEBUG 06:06:02 DEBUG } 06:06:02 DEBUG "value": [ 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 06:06:02 DEBUG } 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG 06:06:02 DEBUG "831672320" 06:06:02 DEBUG "0.1733920414260879" 06:06:02 DEBUG "instance": "10.142.0.51:9100" 06:06:02 DEBUG ] 06:06:02 DEBUG ] 06:06:02 DEBUG }, 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:02 DEBUG "value": [ 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG "1661843.366666667" 06:06:02 DEBUG } 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG { 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG "data": { 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG "result": [ 06:06:02 DEBUG { 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "instance": "10.142.0.11:9100" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG [...] 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "instance": "10.142.0.51:9100" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG "1674243.0333333334" 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735365962" 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG { 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG "data": { 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG "result": [ 06:06:02 DEBUG { 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG [...] 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG "14885847040" 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG { 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG "data": { 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG "result": [ 06:06:02 DEBUG { 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "pod": "ds-idrepo-0" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG [...] 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG "0" 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735365962" 06:06:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735365962" 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG { 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG "data": { 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG "result": [ 06:06:02 DEBUG { 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "pod": "ds-idrepo-0" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG [...] 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG "0.0012158850039442706" 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG { 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG "data": { 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG "result": [ 06:06:02 DEBUG { 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "instance": "10.142.0.11:9100" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG [...] 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "instance": "10.142.0.51:9100" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG "0.00043333333334203425" 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:02 INFO [http_cmd]: http status code OK 06:06:02 DEBUG --- status code --- 06:06:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:02 DEBUG --- http response --- 06:06:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:06:02 DEBUG ----- output ----- 06:06:02 DEBUG { 06:06:02 DEBUG "status": "success", 06:06:02 DEBUG "data": { 06:06:02 DEBUG "resultType": "vector", 06:06:02 DEBUG "result": [ 06:06:02 DEBUG { 06:06:02 DEBUG "metric": { 06:06:02 DEBUG "pod": "ds-idrepo-0" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG [...] 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG 06:06:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:06:02 DEBUG }, 06:06:02 DEBUG "value": [ 06:06:02 DEBUG 1735365962, 06:06:02 DEBUG "542.1751008861829" 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG ] 06:06:02 DEBUG } 06:06:02 DEBUG } 06:06:35 INFO 06:06:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:06:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:06:35 INFO [loop_until]: OK (rc = 0) 06:06:35 DEBUG --- stdout --- 06:06:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 122m 13383Mi rcs-687d95c5b-qs2xv 106m 426Mi rcs-687d95c5b-xxxmk 2m 395Mi 06:06:35 DEBUG --- stderr --- 06:06:35 DEBUG 06:06:37 INFO 06:06:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:06:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:06:37 INFO [loop_until]: OK (rc = 0) 06:06:37 DEBUG --- stdout --- 06:06:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 244m 1% 3419Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 224m 0% 18873Mi 66% 06:06:37 DEBUG --- stderr --- 06:06:37 DEBUG 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 ERROR [http_cmd]: ERROR 06:07:02 ERROR --- status code --- 06:07:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:07:02 ERROR --- http response --- 06:07:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:07:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 DEBUG --- status code --- 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 DEBUG --- status code --- 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG "data": { 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG "result": [ 06:07:02 DEBUG { 06:07:02 DEBUG "metric": { 06:07:02 DEBUG "instance": "10.142.0.11:9100" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG [...] 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG "instance": "10.142.0.51:9100" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG "28808.533333333336" 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG "data": { 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:07:02 DEBUG "result": [ 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG "metric": { 06:07:02 DEBUG "pod": "ds-idrepo-0" 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG { 06:07:02 DEBUG }, 06:07:02 DEBUG "data": { 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG "value": [ 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG "result": [ 06:07:02 DEBUG 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG "data": { 06:07:02 DEBUG { 06:07:02 DEBUG 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG "metric": { 06:07:02 DEBUG 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG "result": [ 06:07:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:07:02 DEBUG [...] 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG { 06:07:02 DEBUG }, 06:07:02 DEBUG 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:07:02 DEBUG "metric": { 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:07:02 DEBUG "instance": "10.142.0.11:9100" 06:07:02 DEBUG 06:07:02 DEBUG }, 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG }, 06:07:02 DEBUG 06:07:02 DEBUG "value": [ 06:07:02 DEBUG "data": { 06:07:02 DEBUG "value": [ 06:07:02 DEBUG [...] 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG "831672320" 06:07:02 DEBUG "result": [ 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG ] 06:07:02 DEBUG { 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG } 06:07:02 DEBUG "metric": { 06:07:02 DEBUG [...] 06:07:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG "0.15533747626492356" 06:07:02 DEBUG "instance": "10.142.0.11:9100" 06:07:02 DEBUG 06:07:02 DEBUG ] 06:07:02 DEBUG ] 06:07:02 DEBUG }, 06:07:02 DEBUG 06:07:02 DEBUG } 06:07:02 DEBUG } 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 06:07:02 DEBUG } 06:07:02 DEBUG ] 06:07:02 DEBUG 06:07:02 DEBUG "instance": "10.142.0.51:9100" 06:07:02 DEBUG } 06:07:02 DEBUG 06:07:02 DEBUG }, 06:07:02 DEBUG } 06:07:02 DEBUG 06:07:02 DEBUG "value": [ 06:07:02 DEBUG [...] 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG 06:07:02 DEBUG "0.0006666666666665342" 06:07:02 DEBUG 06:07:02 DEBUG ] 06:07:02 DEBUG 06:07:02 DEBUG } 06:07:02 DEBUG "instance": "10.142.0.51:9100" 06:07:02 DEBUG ] 06:07:02 DEBUG }, 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG } 06:07:02 DEBUG "value": [ 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG } 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG "1214119.4000000004" 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG ] 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:07:02 DEBUG } 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG ] 06:07:02 DEBUG { 06:07:02 DEBUG } 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG } 06:07:02 DEBUG "data": { 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG "result": [ 06:07:02 DEBUG { 06:07:02 DEBUG "metric": { 06:07:02 DEBUG "instance": "10.142.0.11:9100" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG [...] 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG "instance": "10.142.0.51:9100" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG "1231521.166666667" 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG } 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG "data": { 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG "result": [ 06:07:02 DEBUG { 06:07:02 DEBUG "metric": { 06:07:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG [...] 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG "14882058240" 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG "data": { 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG "result": [ 06:07:02 DEBUG { 06:07:02 DEBUG "metric": { 06:07:02 DEBUG "pod": "ds-idrepo-0" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG [...] 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG "0" 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG } 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366022" 06:07:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366022" 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG "data": { 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG "result": [ 06:07:02 DEBUG { 06:07:02 DEBUG "metric": { 06:07:02 DEBUG "pod": "ds-idrepo-0" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG [...] 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG "0.0013284023059996375" 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG "data": { 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG "result": [ 06:07:02 DEBUG { 06:07:02 DEBUG "metric": { 06:07:02 DEBUG "instance": "10.142.0.11:9100" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG [...] 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG "instance": "10.142.0.51:9100" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG "0.0013333333333311736" 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG } 06:07:02 INFO [http_cmd]: http status code OK 06:07:02 DEBUG --- status code --- 06:07:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:02 DEBUG --- http response --- 06:07:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:07:02 DEBUG ----- output ----- 06:07:02 DEBUG { 06:07:02 DEBUG "status": "success", 06:07:02 DEBUG "data": { 06:07:02 DEBUG "resultType": "vector", 06:07:02 DEBUG "result": [ 06:07:02 DEBUG { 06:07:02 DEBUG "metric": { 06:07:02 DEBUG "pod": "ds-idrepo-0" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG [...] 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG 06:07:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:07:02 DEBUG }, 06:07:02 DEBUG "value": [ 06:07:02 DEBUG 1735366022, 06:07:02 DEBUG "778.982537818009" 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG ] 06:07:02 DEBUG } 06:07:02 DEBUG } 06:07:35 INFO 06:07:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:07:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:07:35 INFO [loop_until]: OK (rc = 0) 06:07:35 DEBUG --- stdout --- 06:07:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 59m 13383Mi rcs-687d95c5b-qs2xv 56m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:07:35 DEBUG --- stderr --- 06:07:35 DEBUG 06:07:37 INFO 06:07:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:07:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:07:37 INFO [loop_until]: OK (rc = 0) 06:07:37 DEBUG --- stdout --- 06:07:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 176m 1% 3418Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 149m 0% 18878Mi 66% 06:07:37 DEBUG --- stderr --- 06:07:37 DEBUG 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 ERROR [http_cmd]: ERROR 06:08:02 DEBUG --- status code --- 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG --- status code --- 06:08:02 ERROR --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG --- http response --- 06:08:02 ERROR --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG { 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG "data": { 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG "result": [ 06:08:02 DEBUG { 06:08:02 DEBUG "metric": { 06:08:02 DEBUG "instance": "10.142.0.11:9100" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG [...] 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG "instance": "10.142.0.51:9100" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG "19251.200000000004" 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366082" 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG { 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG "data": { 06:08:02 DEBUG { 06:08:02 DEBUG { 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG "result": [ 06:08:02 DEBUG "data": { 06:08:02 DEBUG "data": { 06:08:02 DEBUG { 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG "metric": { 06:08:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:08:02 DEBUG "result": [ 06:08:02 DEBUG }, 06:08:02 DEBUG "result": [ 06:08:02 DEBUG { 06:08:02 DEBUG "value": [ 06:08:02 DEBUG { 06:08:02 DEBUG "metric": { 06:08:02 DEBUG 06:08:02 DEBUG "metric": { 06:08:02 DEBUG "pod": "ds-idrepo-0" 06:08:02 DEBUG 06:08:02 DEBUG "instance": "10.142.0.11:9100" 06:08:02 DEBUG }, 06:08:02 DEBUG 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG [...] 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG 06:08:02 DEBUG [...] 06:08:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG [...] 06:08:02 DEBUG 06:08:02 DEBUG }, 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG "value": [ 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG 06:08:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:08:02 DEBUG "0.09850349910868984" 06:08:02 DEBUG { 06:08:02 DEBUG { 06:08:02 DEBUG "instance": "10.142.0.51:9100" 06:08:02 DEBUG }, 06:08:02 DEBUG ] 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG } 06:08:02 DEBUG "data": { 06:08:02 DEBUG "data": { 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG ] 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG "831672320" 06:08:02 DEBUG } 06:08:02 DEBUG "result": [ 06:08:02 DEBUG "result": [ 06:08:02 DEBUG "0.0006666666666674815" 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG { 06:08:02 DEBUG { 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG "metric": { 06:08:02 DEBUG "metric": { 06:08:02 DEBUG } 06:08:02 DEBUG ] 06:08:02 DEBUG "instance": "10.142.0.11:9100" 06:08:02 DEBUG "instance": "10.142.0.11:9100" 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG }, 06:08:02 DEBUG }, 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:02 DEBUG "value": [ 06:08:02 DEBUG "value": [ 06:08:02 DEBUG } 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG [...] 06:08:02 DEBUG [...] 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG "instance": "10.142.0.51:9100" 06:08:02 DEBUG "instance": "10.142.0.51:9100" 06:08:02 DEBUG }, 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG "2354908.233333334" 06:08:02 DEBUG "2337952.2333333343" 06:08:02 DEBUG ] 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:02 DEBUG ] 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG { 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG "data": { 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG "result": [ 06:08:02 DEBUG { 06:08:02 DEBUG "metric": { 06:08:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG [...] 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG "14888579072" 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG { 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG "data": { 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG "result": [ 06:08:02 DEBUG { 06:08:02 DEBUG "metric": { 06:08:02 DEBUG "pod": "ds-idrepo-0" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG [...] 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG "0" 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366082" 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366082" 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG { 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG "data": { 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG "result": [ 06:08:02 DEBUG { 06:08:02 DEBUG "metric": { 06:08:02 DEBUG "pod": "ds-idrepo-0" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG [...] 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG "0.000935875850383233" 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:02 INFO [http_cmd]: http status code OK 06:08:02 DEBUG --- status code --- 06:08:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:02 DEBUG --- http response --- 06:08:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:08:02 DEBUG ----- output ----- 06:08:02 DEBUG { 06:08:02 DEBUG "status": "success", 06:08:02 DEBUG "data": { 06:08:02 DEBUG "resultType": "vector", 06:08:02 DEBUG "result": [ 06:08:02 DEBUG { 06:08:02 DEBUG "metric": { 06:08:02 DEBUG "instance": "10.142.0.11:9100" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG [...] 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG 06:08:02 DEBUG "instance": "10.142.0.51:9100" 06:08:02 DEBUG }, 06:08:02 DEBUG "value": [ 06:08:02 DEBUG 1735366082, 06:08:02 DEBUG "0.0003666666666655753" 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG ] 06:08:02 DEBUG } 06:08:02 DEBUG } 06:08:03 INFO [http_cmd]: http status code OK 06:08:03 DEBUG --- status code --- 06:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:03 DEBUG --- http response --- 06:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:08:03 DEBUG ----- output ----- 06:08:03 DEBUG { 06:08:03 DEBUG "status": "success", 06:08:03 DEBUG "data": { 06:08:03 DEBUG "resultType": "vector", 06:08:03 DEBUG "result": [ 06:08:03 DEBUG { 06:08:03 DEBUG "metric": { 06:08:03 DEBUG "pod": "ds-idrepo-0" 06:08:03 DEBUG }, 06:08:03 DEBUG "value": [ 06:08:03 DEBUG 06:08:03 DEBUG 06:08:03 DEBUG 06:08:03 DEBUG [...] 06:08:03 DEBUG 06:08:03 DEBUG 06:08:03 DEBUG 06:08:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:08:03 DEBUG }, 06:08:03 DEBUG "value": [ 06:08:03 DEBUG 1735366082, 06:08:03 DEBUG "0" 06:08:03 DEBUG ] 06:08:03 DEBUG } 06:08:03 DEBUG ] 06:08:03 DEBUG } 06:08:03 DEBUG } 06:08:35 INFO 06:08:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:08:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:08:35 INFO [loop_until]: OK (rc = 0) 06:08:35 DEBUG --- stdout --- 06:08:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 121m 13382Mi rcs-687d95c5b-qs2xv 130m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:08:35 DEBUG --- stderr --- 06:08:35 DEBUG 06:08:37 INFO 06:08:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:08:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:08:37 INFO [loop_until]: OK (rc = 0) 06:08:37 DEBUG --- stdout --- 06:08:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 197m 1% 3425Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 143m 0% 18883Mi 66% 06:08:37 DEBUG --- stderr --- 06:08:37 DEBUG 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 ERROR [http_cmd]: ERROR 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 ERROR --- status code --- 06:09:02 DEBUG --- status code --- 06:09:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 ERROR --- http response --- 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 DEBUG --- status code --- 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 DEBUG --- http response --- 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG --- status code --- 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:09:02 DEBUG ----- output ----- 06:09:02 DEBUG { 06:09:02 DEBUG "status": "success", 06:09:02 DEBUG "data": { 06:09:02 DEBUG "resultType": "vector", 06:09:02 DEBUG "result": [ 06:09:02 DEBUG { 06:09:02 DEBUG "metric": { 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG "instance": "10.142.0.11:9100" 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG }, 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG "value": [ 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG 06:09:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG [...] 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG "instance": "10.142.0.51:9100" 06:09:02 DEBUG }, 06:09:02 DEBUG "value": [ 06:09:02 DEBUG 1735366142, 06:09:02 DEBUG "25395.200000000004" 06:09:02 DEBUG ] 06:09:02 DEBUG } 06:09:02 DEBUG ] 06:09:02 DEBUG } 06:09:02 DEBUG } 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366142" 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:09:02 DEBUG ----- output ----- 06:09:02 DEBUG { 06:09:02 DEBUG "status": "success", 06:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:09:02 DEBUG "data": { 06:09:02 DEBUG ----- output ----- 06:09:02 DEBUG "resultType": "vector", 06:09:02 DEBUG { 06:09:02 DEBUG "result": [ 06:09:02 DEBUG "status": "success", 06:09:02 DEBUG { 06:09:02 DEBUG "data": { 06:09:02 DEBUG "metric": { 06:09:02 DEBUG "resultType": "vector", 06:09:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:09:02 DEBUG "result": [ 06:09:02 DEBUG }, 06:09:02 DEBUG { 06:09:02 DEBUG "value": [ 06:09:02 DEBUG 06:09:02 DEBUG "metric": { 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG 06:09:02 DEBUG "pod": "ds-idrepo-0" 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG 06:09:02 DEBUG }, 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG [...] 06:09:02 DEBUG "value": [ 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG ----- output ----- 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG { 06:09:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:09:02 DEBUG [...] 06:09:02 DEBUG "status": "success", 06:09:02 DEBUG }, 06:09:02 DEBUG 06:09:02 DEBUG "data": { 06:09:02 DEBUG "value": [ 06:09:02 DEBUG 06:09:02 DEBUG "resultType": "vector", 06:09:02 DEBUG 1735366142, 06:09:02 DEBUG 06:09:02 DEBUG "result": [ 06:09:02 DEBUG "0.05316361378480768" 06:09:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:09:02 DEBUG { 06:09:02 DEBUG ] 06:09:02 DEBUG }, 06:09:02 DEBUG "metric": { 06:09:02 DEBUG } 06:09:02 DEBUG "value": [ 06:09:02 DEBUG "instance": "10.142.0.11:9100" 06:09:02 DEBUG ] 06:09:02 DEBUG 1735366142, 06:09:02 DEBUG }, 06:09:02 DEBUG } 06:09:02 DEBUG "831672320" 06:09:02 DEBUG "value": [ 06:09:02 DEBUG } 06:09:02 DEBUG ] 06:09:02 DEBUG 06:09:02 DEBUG } 06:09:02 DEBUG 06:09:02 DEBUG ] 06:09:02 DEBUG 06:09:02 DEBUG } 06:09:02 DEBUG [...] 06:09:02 DEBUG } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG "instance": "10.142.0.51:9100" 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG }, 06:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:09:02 DEBUG "value": [ 06:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:09:02 DEBUG ----- output ----- 06:09:02 DEBUG 1735366142, 06:09:02 DEBUG ----- output ----- 06:09:02 DEBUG { 06:09:02 DEBUG "0.00033333333333303023" 06:09:02 DEBUG { 06:09:02 DEBUG "status": "success", 06:09:02 DEBUG ] 06:09:02 DEBUG "status": "success", 06:09:02 DEBUG "data": { 06:09:02 DEBUG } 06:09:02 DEBUG "data": { 06:09:02 DEBUG "resultType": "vector", 06:09:02 DEBUG ] 06:09:02 DEBUG "resultType": "vector", 06:09:02 DEBUG "result": [ 06:09:02 DEBUG } 06:09:02 DEBUG "result": [ 06:09:02 DEBUG { 06:09:02 DEBUG } 06:09:02 DEBUG { 06:09:02 DEBUG "metric": { 06:09:02 DEBUG "metric": { 06:09:02 DEBUG "instance": "10.142.0.11:9100" 06:09:02 DEBUG "instance": "10.142.0.11:9100" 06:09:02 DEBUG }, 06:09:02 DEBUG }, 06:09:02 DEBUG "value": [ 06:09:02 DEBUG "value": [ 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG [...] 06:09:02 DEBUG [...] 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG "instance": "10.142.0.51:9100" 06:09:02 DEBUG "instance": "10.142.0.51:9100" 06:09:02 DEBUG }, 06:09:02 DEBUG }, 06:09:02 DEBUG "value": [ 06:09:02 DEBUG "value": [ 06:09:02 DEBUG 1735366142, 06:09:02 DEBUG 1735366142, 06:09:02 DEBUG "960901.8333333338" 06:09:02 DEBUG "944193.3666666667" 06:09:02 DEBUG ] 06:09:02 DEBUG ] 06:09:02 DEBUG } 06:09:02 DEBUG } 06:09:02 DEBUG ] 06:09:02 DEBUG ] 06:09:02 DEBUG } 06:09:02 DEBUG } 06:09:02 DEBUG } 06:09:02 DEBUG } 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:09:02 DEBUG ----- output ----- 06:09:02 DEBUG { 06:09:02 DEBUG "status": "success", 06:09:02 DEBUG "data": { 06:09:02 DEBUG "resultType": "vector", 06:09:02 DEBUG "result": [ 06:09:02 DEBUG { 06:09:02 DEBUG "metric": { 06:09:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:09:02 DEBUG }, 06:09:02 DEBUG "value": [ 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG [...] 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:09:02 DEBUG }, 06:09:02 DEBUG "value": [ 06:09:02 DEBUG 1735366142, 06:09:02 DEBUG "14887202816" 06:09:02 DEBUG ] 06:09:02 DEBUG } 06:09:02 DEBUG ] 06:09:02 DEBUG } 06:09:02 DEBUG } 06:09:02 INFO [http_cmd]: http status code OK 06:09:02 DEBUG --- status code --- 06:09:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:02 DEBUG --- http response --- 06:09:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:09:02 DEBUG ----- output ----- 06:09:02 DEBUG { 06:09:02 DEBUG "status": "success", 06:09:02 DEBUG "data": { 06:09:02 DEBUG "resultType": "vector", 06:09:02 DEBUG "result": [ 06:09:02 DEBUG { 06:09:02 DEBUG "metric": { 06:09:02 DEBUG "pod": "ds-cts-0" 06:09:02 DEBUG }, 06:09:02 DEBUG "value": [ 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG [...] 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG 06:09:02 DEBUG "pod": "ds-idrepo-0" 06:09:02 DEBUG }, 06:09:02 DEBUG "value": [ 06:09:02 DEBUG 1735366142, 06:09:02 DEBUG "0" 06:09:02 DEBUG ] 06:09:02 DEBUG } 06:09:02 DEBUG ] 06:09:02 DEBUG } 06:09:02 DEBUG } 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366142" 06:09:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366142" 06:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366143" 06:09:03 INFO [http_cmd]: http status code OK 06:09:03 DEBUG --- status code --- 06:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:03 DEBUG --- http response --- 06:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:09:03 DEBUG ----- output ----- 06:09:03 DEBUG { 06:09:03 DEBUG "status": "success", 06:09:03 DEBUG "data": { 06:09:03 DEBUG "resultType": "vector", 06:09:03 DEBUG "result": [ 06:09:03 DEBUG { 06:09:03 DEBUG "metric": { 06:09:03 DEBUG "instance": "10.142.0.11:9100" 06:09:03 DEBUG }, 06:09:03 DEBUG "value": [ 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG [...] 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG "instance": "10.142.0.51:9100" 06:09:03 DEBUG }, 06:09:03 DEBUG "value": [ 06:09:03 DEBUG 1735366142, 06:09:03 DEBUG "0.00036666666666557537" 06:09:03 DEBUG ] 06:09:03 DEBUG } 06:09:03 DEBUG ] 06:09:03 DEBUG } 06:09:03 DEBUG } 06:09:03 INFO [http_cmd]: http status code OK 06:09:03 DEBUG --- status code --- 06:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:03 DEBUG --- http response --- 06:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:09:03 DEBUG ----- output ----- 06:09:03 DEBUG { 06:09:03 DEBUG "status": "success", 06:09:03 DEBUG "data": { 06:09:03 DEBUG "resultType": "vector", 06:09:03 DEBUG "result": [ 06:09:03 DEBUG { 06:09:03 DEBUG "metric": { 06:09:03 DEBUG "pod": "ds-cts-0" 06:09:03 DEBUG }, 06:09:03 DEBUG "value": [ 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG [...] 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG "pod": "ds-idrepo-0" 06:09:03 DEBUG }, 06:09:03 DEBUG "value": [ 06:09:03 DEBUG 1735366142, 06:09:03 DEBUG "0.006175303020469932" 06:09:03 DEBUG ] 06:09:03 DEBUG } 06:09:03 DEBUG ] 06:09:03 DEBUG } 06:09:03 DEBUG } 06:09:03 INFO [http_cmd]: http status code OK 06:09:03 DEBUG --- status code --- 06:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:03 DEBUG --- http response --- 06:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:09:03 DEBUG ----- output ----- 06:09:03 DEBUG { 06:09:03 DEBUG "status": "success", 06:09:03 DEBUG "data": { 06:09:03 DEBUG "resultType": "vector", 06:09:03 DEBUG "result": [ 06:09:03 DEBUG { 06:09:03 DEBUG "metric": { 06:09:03 DEBUG "pod": "ds-cts-0" 06:09:03 DEBUG }, 06:09:03 DEBUG "value": [ 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG [...] 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG 06:09:03 DEBUG "pod": "ds-idrepo-0" 06:09:03 DEBUG }, 06:09:03 DEBUG "value": [ 06:09:03 DEBUG 1735366143, 06:09:03 DEBUG "919.3869043801707" 06:09:03 DEBUG ] 06:09:03 DEBUG } 06:09:03 DEBUG ] 06:09:03 DEBUG } 06:09:03 DEBUG } 06:09:35 INFO 06:09:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:09:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:09:35 INFO [loop_until]: OK (rc = 0) 06:09:35 DEBUG --- stdout --- 06:09:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 546m 13385Mi rcs-687d95c5b-qs2xv 52m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:09:35 DEBUG --- stderr --- 06:09:35 DEBUG 06:09:37 INFO 06:09:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:09:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:09:37 INFO [loop_until]: OK (rc = 0) 06:09:37 DEBUG --- stdout --- 06:09:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 172m 1% 3455Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 704m 2% 18884Mi 66% 06:09:37 DEBUG --- stderr --- 06:09:37 DEBUG 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 ERROR [http_cmd]: ERROR 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 ERROR --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 ERROR --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:10:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366202" 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG --- status code --- 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:10:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:02 DEBUG ----- output ----- 06:10:02 DEBUG { 06:10:02 DEBUG "status": "success", 06:10:02 DEBUG "data": { 06:10:02 DEBUG "resultType": "vector", 06:10:02 DEBUG "result": [ 06:10:02 DEBUG { 06:10:02 DEBUG "metric": { 06:10:02 DEBUG "instance": "10.142.0.11:9100" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG [...] 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG "instance": "10.142.0.51:9100" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 1735366202, 06:10:02 DEBUG "27989.333333333336" 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG } 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366202" 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:10:02 DEBUG ----- output ----- 06:10:02 DEBUG { 06:10:02 DEBUG "status": "success", 06:10:02 DEBUG "data": { 06:10:02 DEBUG "resultType": "vector", 06:10:02 DEBUG "result": [ 06:10:02 DEBUG { 06:10:02 DEBUG "metric": { 06:10:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG [...] 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 1735366202, 06:10:02 DEBUG "0.6513618333492037" 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG ----- output ----- 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { 06:10:02 DEBUG "status": "success", 06:10:02 DEBUG "data": { 06:10:02 DEBUG "resultType": "vector", 06:10:02 DEBUG "result": [ 06:10:02 DEBUG { 06:10:02 DEBUG "metric": { 06:10:02 DEBUG "pod": "ds-idrepo-0" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG [...] 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 1735366202, 06:10:02 DEBUG "831672320" 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG } 06:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:10:02 DEBUG ----- output ----- 06:10:02 DEBUG { 06:10:02 DEBUG "status": "success", 06:10:02 DEBUG "data": { 06:10:02 DEBUG "resultType": "vector", 06:10:02 DEBUG "result": [ 06:10:02 DEBUG { 06:10:02 DEBUG "metric": { 06:10:02 DEBUG "instance": "10.142.0.11:9100" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG [...] 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG "instance": "10.142.0.51:9100" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 1735366202, 06:10:02 DEBUG "0.0010000000000009853" 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG } 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:10:02 DEBUG ----- output ----- 06:10:02 DEBUG { 06:10:02 DEBUG "status": "success", 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG "data": { 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG "resultType": "vector", 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG "result": [ 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG { 06:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:10:02 DEBUG "metric": { 06:10:02 DEBUG ----- output ----- 06:10:02 DEBUG "instance": "10.142.0.11:9100" 06:10:02 DEBUG { 06:10:02 DEBUG }, 06:10:02 DEBUG "status": "success", 06:10:02 DEBUG "value": [ 06:10:02 DEBUG "data": { 06:10:02 DEBUG 06:10:02 DEBUG "resultType": "vector", 06:10:02 DEBUG 06:10:02 DEBUG "result": [ 06:10:02 DEBUG 06:10:02 DEBUG { 06:10:02 DEBUG [...] 06:10:02 DEBUG "metric": { 06:10:02 DEBUG 06:10:02 DEBUG "instance": "10.142.0.11:9100" 06:10:02 DEBUG 06:10:02 DEBUG }, 06:10:02 DEBUG 06:10:02 DEBUG "value": [ 06:10:02 DEBUG "instance": "10.142.0.51:9100" 06:10:02 DEBUG 06:10:02 DEBUG }, 06:10:02 DEBUG 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 06:10:02 DEBUG 1735366202, 06:10:02 DEBUG [...] 06:10:02 DEBUG "1243877.366666667" 06:10:02 DEBUG 06:10:02 DEBUG ] 06:10:02 DEBUG 06:10:02 DEBUG } 06:10:02 DEBUG 06:10:02 DEBUG ] 06:10:02 DEBUG "instance": "10.142.0.51:9100" 06:10:02 DEBUG } 06:10:02 DEBUG }, 06:10:02 DEBUG } 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 1735366202, 06:10:02 DEBUG "1226368.8333333337" 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG } 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366202" 06:10:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366202" 06:10:02 INFO [http_cmd]: http status code OK 06:10:02 DEBUG --- status code --- 06:10:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:02 DEBUG --- http response --- 06:10:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:10:02 DEBUG ----- output ----- 06:10:02 DEBUG { 06:10:02 DEBUG "status": "success", 06:10:02 DEBUG "data": { 06:10:02 DEBUG "resultType": "vector", 06:10:02 DEBUG "result": [ 06:10:02 DEBUG { 06:10:02 DEBUG "metric": { 06:10:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG [...] 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG 06:10:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:10:02 DEBUG }, 06:10:02 DEBUG "value": [ 06:10:02 DEBUG 1735366202, 06:10:02 DEBUG "14892597248" 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG ] 06:10:02 DEBUG } 06:10:02 DEBUG } 06:10:03 INFO [http_cmd]: http status code OK 06:10:03 DEBUG --- status code --- 06:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:03 DEBUG --- http response --- 06:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:10:03 DEBUG ----- output ----- 06:10:03 DEBUG { 06:10:03 DEBUG "status": "success", 06:10:03 DEBUG "data": { 06:10:03 DEBUG "resultType": "vector", 06:10:03 DEBUG "result": [ 06:10:03 DEBUG { 06:10:03 DEBUG "metric": { 06:10:03 DEBUG "pod": "ds-idrepo-0" 06:10:03 DEBUG }, 06:10:03 DEBUG "value": [ 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG [...] 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:10:03 DEBUG }, 06:10:03 DEBUG "value": [ 06:10:03 DEBUG 1735366202, 06:10:03 DEBUG "0" 06:10:03 DEBUG ] 06:10:03 DEBUG } 06:10:03 DEBUG ] 06:10:03 DEBUG } 06:10:03 DEBUG } 06:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366203" 06:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366203" 06:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366203" 06:10:03 INFO [http_cmd]: http status code OK 06:10:03 INFO [http_cmd]: http status code OK 06:10:03 DEBUG --- status code --- 06:10:03 DEBUG --- status code --- 06:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:03 DEBUG --- http response --- 06:10:03 DEBUG --- http response --- 06:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:10:03 DEBUG ----- output ----- 06:10:03 DEBUG ----- output ----- 06:10:03 DEBUG { 06:10:03 DEBUG { 06:10:03 DEBUG "status": "success", 06:10:03 DEBUG "status": "success", 06:10:03 DEBUG "data": { 06:10:03 DEBUG "data": { 06:10:03 DEBUG "resultType": "vector", 06:10:03 DEBUG "resultType": "vector", 06:10:03 DEBUG "result": [ 06:10:03 DEBUG "result": [ 06:10:03 DEBUG { 06:10:03 DEBUG { 06:10:03 DEBUG "metric": { 06:10:03 DEBUG "metric": { 06:10:03 DEBUG "pod": "ds-idrepo-0" 06:10:03 DEBUG "instance": "10.142.0.11:9100" 06:10:03 DEBUG }, 06:10:03 DEBUG }, 06:10:03 DEBUG "value": [ 06:10:03 DEBUG "value": [ 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG [...] 06:10:03 DEBUG [...] 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:10:03 DEBUG "instance": "10.142.0.51:9100" 06:10:03 DEBUG }, 06:10:03 DEBUG }, 06:10:03 DEBUG "value": [ 06:10:03 DEBUG "value": [ 06:10:03 DEBUG 1735366203, 06:10:03 DEBUG 1735366203, 06:10:03 DEBUG "0.0012179021554345156" 06:10:03 DEBUG "0.0009666666666665456" 06:10:03 DEBUG ] 06:10:03 DEBUG ] 06:10:03 DEBUG } 06:10:03 DEBUG } 06:10:03 DEBUG ] 06:10:03 DEBUG ] 06:10:03 DEBUG } 06:10:03 DEBUG } 06:10:03 DEBUG } 06:10:03 DEBUG } 06:10:03 INFO [http_cmd]: http status code OK 06:10:03 DEBUG --- status code --- 06:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:03 DEBUG --- http response --- 06:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:10:03 DEBUG ----- output ----- 06:10:03 DEBUG { 06:10:03 DEBUG "status": "success", 06:10:03 DEBUG "data": { 06:10:03 DEBUG "resultType": "vector", 06:10:03 DEBUG "result": [ 06:10:03 DEBUG { 06:10:03 DEBUG "metric": { 06:10:03 DEBUG "pod": "ds-idrepo-0" 06:10:03 DEBUG }, 06:10:03 DEBUG "value": [ 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG [...] 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG 06:10:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:10:03 DEBUG }, 06:10:03 DEBUG "value": [ 06:10:03 DEBUG 1735366203, 06:10:03 DEBUG "594.8084951896896" 06:10:03 DEBUG ] 06:10:03 DEBUG } 06:10:03 DEBUG ] 06:10:03 DEBUG } 06:10:03 DEBUG } 06:10:35 INFO 06:10:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:10:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:10:35 INFO [loop_until]: OK (rc = 0) 06:10:35 DEBUG --- stdout --- 06:10:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 67m 13386Mi rcs-687d95c5b-qs2xv 114m 427Mi rcs-687d95c5b-xxxmk 2m 395Mi 06:10:35 DEBUG --- stderr --- 06:10:35 DEBUG 06:10:37 INFO 06:10:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:10:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:10:37 INFO [loop_until]: OK (rc = 0) 06:10:37 DEBUG --- stdout --- 06:10:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 182m 1% 3468Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 144m 0% 18887Mi 66% 06:10:37 DEBUG --- stderr --- 06:10:37 DEBUG 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 ERROR [http_cmd]: ERROR 06:11:02 DEBUG --- status code --- 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- status code --- 06:11:02 ERROR --- status code --- 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- http response --- 06:11:02 ERROR --- http response --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG --- http response --- 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- status code --- 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- status code --- 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG ----- output ----- 06:11:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:02 DEBUG { 06:11:02 DEBUG "status": "success", 06:11:02 DEBUG "data": { 06:11:02 DEBUG "resultType": "vector", 06:11:02 DEBUG "result": [ 06:11:02 DEBUG { 06:11:02 DEBUG "metric": { 06:11:02 DEBUG "instance": "10.142.0.11:9100" 06:11:02 DEBUG }, 06:11:02 DEBUG "value": [ 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG [...] 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG "instance": "10.142.0.51:9100" 06:11:02 DEBUG }, 06:11:02 DEBUG "value": [ 06:11:02 DEBUG 1735366262, 06:11:02 DEBUG "22528.000000000004" 06:11:02 DEBUG ] 06:11:02 DEBUG } 06:11:02 DEBUG ] 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366262" 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366262" 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:11:02 DEBUG ----- output ----- 06:11:02 DEBUG { 06:11:02 DEBUG "status": "success", 06:11:02 DEBUG "data": { 06:11:02 DEBUG "resultType": "vector", 06:11:02 DEBUG "result": [ 06:11:02 DEBUG { 06:11:02 DEBUG "metric": { 06:11:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:11:02 DEBUG }, 06:11:02 DEBUG "value": [ 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG [...] 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:11:02 DEBUG }, 06:11:02 DEBUG "value": [ 06:11:02 DEBUG 1735366262, 06:11:02 DEBUG "0.1490250695216759" 06:11:02 DEBUG ] 06:11:02 DEBUG } 06:11:02 DEBUG ] 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:11:02 DEBUG ----- output ----- 06:11:02 DEBUG ----- output ----- 06:11:02 DEBUG { 06:11:02 DEBUG { 06:11:02 DEBUG "status": "success", 06:11:02 DEBUG "status": "success", 06:11:02 DEBUG "data": { 06:11:02 DEBUG "data": { 06:11:02 DEBUG "resultType": "vector", 06:11:02 DEBUG "resultType": "vector", 06:11:02 DEBUG "result": [ 06:11:02 DEBUG "result": [ 06:11:02 DEBUG { 06:11:02 DEBUG { 06:11:02 DEBUG "metric": { 06:11:02 DEBUG "metric": { 06:11:02 DEBUG "instance": "10.142.0.11:9100" 06:11:02 DEBUG "pod": "ds-idrepo-0" 06:11:02 DEBUG }, 06:11:02 DEBUG }, 06:11:02 DEBUG "value": [ 06:11:02 DEBUG "value": [ 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG [...] 06:11:02 DEBUG [...] 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG "instance": "10.142.0.51:9100" 06:11:02 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:11:02 DEBUG }, 06:11:02 DEBUG }, 06:11:02 DEBUG "value": [ 06:11:02 DEBUG "value": [ 06:11:02 DEBUG 1735366262, 06:11:02 DEBUG 1735366262, 06:11:02 DEBUG "0.0010000000000000382" 06:11:02 DEBUG "831852544" 06:11:02 DEBUG ] 06:11:02 DEBUG ] 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 DEBUG ] 06:11:02 DEBUG ] 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 INFO [http_cmd]: http status code OK 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG --- status code --- 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG --- http response --- 06:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:11:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:11:02 DEBUG ----- output ----- 06:11:02 DEBUG ----- output ----- 06:11:02 DEBUG { 06:11:02 DEBUG { 06:11:02 DEBUG "status": "success", 06:11:02 DEBUG "status": "success", 06:11:02 DEBUG "data": { 06:11:02 DEBUG "data": { 06:11:02 DEBUG "resultType": "vector", 06:11:02 DEBUG "resultType": "vector", 06:11:02 DEBUG "result": [ 06:11:02 DEBUG "result": [ 06:11:02 DEBUG { 06:11:02 DEBUG { 06:11:02 DEBUG "metric": { 06:11:02 DEBUG "metric": { 06:11:02 DEBUG "instance": "10.142.0.11:9100" 06:11:02 DEBUG "instance": "10.142.0.11:9100" 06:11:02 DEBUG }, 06:11:02 DEBUG }, 06:11:02 DEBUG "value": [ 06:11:02 DEBUG "value": [ 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG [...] 06:11:02 DEBUG [...] 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG 06:11:02 DEBUG "instance": "10.142.0.51:9100" 06:11:02 DEBUG "instance": "10.142.0.51:9100" 06:11:02 DEBUG }, 06:11:02 DEBUG }, 06:11:02 DEBUG "value": [ 06:11:02 DEBUG "value": [ 06:11:02 DEBUG 1735366262, 06:11:02 DEBUG 1735366262, 06:11:02 DEBUG "1248120.366666667" 06:11:02 DEBUG "1261084.033333334" 06:11:02 DEBUG ] 06:11:02 DEBUG ] 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 DEBUG ] 06:11:02 DEBUG ] 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 DEBUG } 06:11:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366262" 06:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366263" 06:11:03 INFO [http_cmd]: http status code OK 06:11:03 DEBUG --- status code --- 06:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:03 DEBUG --- http response --- 06:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:11:03 DEBUG ----- output ----- 06:11:03 DEBUG { 06:11:03 DEBUG "status": "success", 06:11:03 DEBUG "data": { 06:11:03 DEBUG "resultType": "vector", 06:11:03 DEBUG "result": [ 06:11:03 DEBUG { 06:11:03 DEBUG "metric": { 06:11:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:11:03 DEBUG }, 06:11:03 DEBUG "value": [ 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG [...] 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:11:03 DEBUG }, 06:11:03 DEBUG "value": [ 06:11:03 DEBUG 1735366262, 06:11:03 DEBUG "14893240320" 06:11:03 DEBUG ] 06:11:03 DEBUG } 06:11:03 DEBUG ] 06:11:03 DEBUG } 06:11:03 DEBUG } 06:11:03 INFO [http_cmd]: http status code OK 06:11:03 DEBUG --- status code --- 06:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:03 DEBUG --- http response --- 06:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:11:03 DEBUG ----- output ----- 06:11:03 DEBUG { 06:11:03 DEBUG "status": "success", 06:11:03 DEBUG "data": { 06:11:03 DEBUG "resultType": "vector", 06:11:03 DEBUG "result": [ 06:11:03 DEBUG { 06:11:03 DEBUG "metric": { 06:11:03 DEBUG "pod": "ds-idrepo-0" 06:11:03 DEBUG }, 06:11:03 DEBUG "value": [ 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG [...] 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:11:03 DEBUG }, 06:11:03 DEBUG "value": [ 06:11:03 DEBUG 1735366263, 06:11:03 DEBUG "0" 06:11:03 DEBUG ] 06:11:03 DEBUG } 06:11:03 DEBUG ] 06:11:03 DEBUG } 06:11:03 DEBUG } 06:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366263" 06:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366263" 06:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366263" 06:11:03 INFO [http_cmd]: http status code OK 06:11:03 DEBUG --- status code --- 06:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:03 DEBUG --- http response --- 06:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:11:03 DEBUG ----- output ----- 06:11:03 DEBUG { 06:11:03 DEBUG "status": "success", 06:11:03 DEBUG "data": { 06:11:03 DEBUG "resultType": "vector", 06:11:03 DEBUG "result": [ 06:11:03 DEBUG { 06:11:03 DEBUG "metric": { 06:11:03 DEBUG "pod": "ds-idrepo-0" 06:11:03 DEBUG }, 06:11:03 DEBUG "value": [ 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG [...] 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:11:03 DEBUG }, 06:11:03 DEBUG "value": [ 06:11:03 DEBUG 1735366263, 06:11:03 DEBUG "0.0015104744359813998" 06:11:03 DEBUG ] 06:11:03 DEBUG } 06:11:03 DEBUG ] 06:11:03 DEBUG } 06:11:03 DEBUG } 06:11:03 INFO [http_cmd]: http status code OK 06:11:03 INFO [http_cmd]: http status code OK 06:11:03 DEBUG --- status code --- 06:11:03 DEBUG --- status code --- 06:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:03 DEBUG --- http response --- 06:11:03 DEBUG --- http response --- 06:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:11:03 DEBUG ----- output ----- 06:11:03 DEBUG ----- output ----- 06:11:03 DEBUG { 06:11:03 DEBUG { 06:11:03 DEBUG "status": "success", 06:11:03 DEBUG "status": "success", 06:11:03 DEBUG "data": { 06:11:03 DEBUG "data": { 06:11:03 DEBUG "resultType": "vector", 06:11:03 DEBUG "resultType": "vector", 06:11:03 DEBUG "result": [ 06:11:03 DEBUG "result": [ 06:11:03 DEBUG { 06:11:03 DEBUG { 06:11:03 DEBUG "metric": { 06:11:03 DEBUG "metric": { 06:11:03 DEBUG "pod": "ds-idrepo-0" 06:11:03 DEBUG "instance": "10.142.0.11:9100" 06:11:03 DEBUG }, 06:11:03 DEBUG }, 06:11:03 DEBUG "value": [ 06:11:03 DEBUG "value": [ 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG [...] 06:11:03 DEBUG [...] 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG 06:11:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:11:03 DEBUG "instance": "10.142.0.51:9100" 06:11:03 DEBUG }, 06:11:03 DEBUG }, 06:11:03 DEBUG "value": [ 06:11:03 DEBUG "value": [ 06:11:03 DEBUG 1735366263, 06:11:03 DEBUG 1735366263, 06:11:03 DEBUG "669.6689533872132" 06:11:03 DEBUG "0.0012000000000038352" 06:11:03 DEBUG ] 06:11:03 DEBUG ] 06:11:03 DEBUG } 06:11:03 DEBUG } 06:11:03 DEBUG ] 06:11:03 DEBUG ] 06:11:03 DEBUG } 06:11:03 DEBUG } 06:11:03 DEBUG } 06:11:03 DEBUG } 06:11:35 INFO 06:11:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:11:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:11:35 INFO [loop_until]: OK (rc = 0) 06:11:35 DEBUG --- stdout --- 06:11:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 65m 13385Mi rcs-687d95c5b-qs2xv 112m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:11:35 DEBUG --- stderr --- 06:11:35 DEBUG 06:11:37 INFO 06:11:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:11:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:11:37 INFO [loop_until]: OK (rc = 0) 06:11:37 DEBUG --- stdout --- 06:11:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 234m 1% 3455Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 201m 0% 18885Mi 66% 06:11:37 DEBUG --- stderr --- 06:11:37 DEBUG 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 ERROR [http_cmd]: ERROR 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 ERROR --- status code --- 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG --- status code --- 06:12:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 ERROR --- http response --- 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG --- http response --- 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366322" 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:12:02 DEBUG ----- output ----- 06:12:02 DEBUG { 06:12:02 DEBUG "status": "success", 06:12:02 DEBUG "data": { 06:12:02 DEBUG "resultType": "vector", 06:12:02 DEBUG "result": [ 06:12:02 DEBUG { 06:12:02 DEBUG "metric": { 06:12:02 DEBUG "instance": "10.142.0.11:9100" 06:12:02 DEBUG }, 06:12:02 DEBUG "value": [ 06:12:02 DEBUG 06:12:02 DEBUG 06:12:02 DEBUG 06:12:02 DEBUG [...] 06:12:02 DEBUG 06:12:02 DEBUG 06:12:02 DEBUG 06:12:02 DEBUG "instance": "10.142.0.51:9100" 06:12:02 DEBUG }, 06:12:02 DEBUG "value": [ 06:12:02 DEBUG 1735366322, 06:12:02 DEBUG "39731.200000000004" 06:12:02 DEBUG ] 06:12:02 DEBUG } 06:12:02 DEBUG ] 06:12:02 DEBUG } 06:12:02 DEBUG } 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366322" 06:12:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366322" 06:12:02 INFO [http_cmd]: http status code OK 06:12:02 DEBUG --- status code --- 06:12:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:02 DEBUG --- http response --- 06:12:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:12:02 DEBUG ----- output ----- 06:12:02 DEBUG { 06:12:02 DEBUG "status": "success", 06:12:02 DEBUG "data": { 06:12:02 DEBUG "resultType": "vector", 06:12:02 DEBUG "result": [ 06:12:02 DEBUG { 06:12:02 DEBUG "metric": { 06:12:02 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:12:02 DEBUG }, 06:12:02 DEBUG "value": [ 06:12:02 DEBUG 06:12:02 DEBUG 06:12:02 DEBUG 06:12:02 DEBUG [...] 06:12:02 DEBUG 06:12:02 DEBUG 06:12:02 DEBUG 06:12:02 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:12:02 DEBUG }, 06:12:02 DEBUG "value": [ 06:12:02 DEBUG 1735366322, 06:12:02 DEBUG "0.14459400957335627" 06:12:02 DEBUG ] 06:12:02 DEBUG } 06:12:02 DEBUG ] 06:12:02 DEBUG } 06:12:02 DEBUG } 06:12:03 INFO [http_cmd]: http status code OK 06:12:03 INFO [http_cmd]: http status code OK 06:12:03 DEBUG --- status code --- 06:12:03 DEBUG --- status code --- 06:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:03 DEBUG --- http response --- 06:12:03 DEBUG --- http response --- 06:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:12:03 DEBUG ----- output ----- 06:12:03 DEBUG ----- output ----- 06:12:03 DEBUG { 06:12:03 DEBUG { 06:12:03 DEBUG "status": "success", 06:12:03 DEBUG "status": "success", 06:12:03 DEBUG "data": { 06:12:03 DEBUG "data": { 06:12:03 DEBUG "resultType": "vector", 06:12:03 DEBUG "resultType": "vector", 06:12:03 DEBUG "result": [ 06:12:03 DEBUG "result": [ 06:12:03 DEBUG { 06:12:03 DEBUG { 06:12:03 DEBUG "metric": { 06:12:03 DEBUG "metric": { 06:12:03 DEBUG "pod": "ds-idrepo-0" 06:12:03 DEBUG "instance": "10.142.0.11:9100" 06:12:03 DEBUG }, 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG [...] 06:12:03 DEBUG [...] 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:12:03 DEBUG "instance": "10.142.0.51:9100" 06:12:03 DEBUG }, 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 1735366322, 06:12:03 DEBUG 1735366322, 06:12:03 DEBUG "831672320" 06:12:03 DEBUG "0.0010000000000000382" 06:12:03 DEBUG ] 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 DEBUG ] 06:12:03 INFO [http_cmd]: http status code OK 06:12:03 INFO [http_cmd]: http status code OK 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG --- status code --- 06:12:03 DEBUG --- status code --- 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:03 DEBUG } 06:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366323" 06:12:03 DEBUG --- http response --- 06:12:03 DEBUG --- http response --- 06:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:12:03 DEBUG ----- output ----- 06:12:03 DEBUG ----- output ----- 06:12:03 DEBUG { 06:12:03 DEBUG { 06:12:03 DEBUG "status": "success", 06:12:03 DEBUG "status": "success", 06:12:03 DEBUG "data": { 06:12:03 DEBUG "data": { 06:12:03 DEBUG "resultType": "vector", 06:12:03 DEBUG "resultType": "vector", 06:12:03 DEBUG "result": [ 06:12:03 DEBUG "result": [ 06:12:03 DEBUG { 06:12:03 DEBUG { 06:12:03 DEBUG "metric": { 06:12:03 DEBUG "metric": { 06:12:03 DEBUG "instance": "10.142.0.11:9100" 06:12:03 DEBUG "instance": "10.142.0.11:9100" 06:12:03 DEBUG }, 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG [...] 06:12:03 DEBUG [...] 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG "instance": "10.142.0.51:9100" 06:12:03 DEBUG "instance": "10.142.0.51:9100" 06:12:03 DEBUG }, 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 1735366322, 06:12:03 DEBUG 1735366322, 06:12:03 DEBUG "2173063.133333334" 06:12:03 DEBUG "2156369.9333333345" 06:12:03 DEBUG ] 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 DEBUG ] 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366323" 06:12:03 INFO [http_cmd]: http status code OK 06:12:03 DEBUG --- status code --- 06:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:03 DEBUG --- http response --- 06:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:12:03 DEBUG ----- output ----- 06:12:03 DEBUG { 06:12:03 DEBUG "status": "success", 06:12:03 DEBUG "data": { 06:12:03 DEBUG "resultType": "vector", 06:12:03 DEBUG "result": [ 06:12:03 DEBUG { 06:12:03 DEBUG "metric": { 06:12:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG [...] 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 1735366323, 06:12:03 DEBUG "14892515328" 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 INFO [http_cmd]: http status code OK 06:12:03 DEBUG --- status code --- 06:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:03 DEBUG --- http response --- 06:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:12:03 DEBUG ----- output ----- 06:12:03 DEBUG { 06:12:03 DEBUG "status": "success", 06:12:03 DEBUG "data": { 06:12:03 DEBUG "resultType": "vector", 06:12:03 DEBUG "result": [ 06:12:03 DEBUG { 06:12:03 DEBUG "metric": { 06:12:03 DEBUG "pod": "ds-idrepo-0" 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG [...] 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 1735366323, 06:12:03 DEBUG "0" 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366323" 06:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366323" 06:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366323" 06:12:03 INFO [http_cmd]: http status code OK 06:12:03 DEBUG --- status code --- 06:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:03 DEBUG --- http response --- 06:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:12:03 DEBUG ----- output ----- 06:12:03 DEBUG { 06:12:03 DEBUG "status": "success", 06:12:03 DEBUG "data": { 06:12:03 DEBUG "resultType": "vector", 06:12:03 DEBUG "result": [ 06:12:03 DEBUG { 06:12:03 DEBUG "metric": { 06:12:03 DEBUG "pod": "ds-idrepo-0" 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG [...] 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 1735366323, 06:12:03 DEBUG "0.0012648234275600915" 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 INFO [http_cmd]: http status code OK 06:12:03 INFO [http_cmd]: http status code OK 06:12:03 DEBUG --- status code --- 06:12:03 DEBUG --- status code --- 06:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:03 DEBUG --- http response --- 06:12:03 DEBUG --- http response --- 06:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:12:03 DEBUG ----- output ----- 06:12:03 DEBUG ----- output ----- 06:12:03 DEBUG { 06:12:03 DEBUG { 06:12:03 DEBUG "status": "success", 06:12:03 DEBUG "status": "success", 06:12:03 DEBUG "data": { 06:12:03 DEBUG "data": { 06:12:03 DEBUG "resultType": "vector", 06:12:03 DEBUG "resultType": "vector", 06:12:03 DEBUG "result": [ 06:12:03 DEBUG "result": [ 06:12:03 DEBUG { 06:12:03 DEBUG { 06:12:03 DEBUG "metric": { 06:12:03 DEBUG "metric": { 06:12:03 DEBUG "instance": "10.142.0.11:9100" 06:12:03 DEBUG "pod": "ds-idrepo-0" 06:12:03 DEBUG }, 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG [...] 06:12:03 DEBUG [...] 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG 06:12:03 DEBUG "instance": "10.142.0.51:9100" 06:12:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:12:03 DEBUG }, 06:12:03 DEBUG }, 06:12:03 DEBUG "value": [ 06:12:03 DEBUG "value": [ 06:12:03 DEBUG 1735366323, 06:12:03 DEBUG 1735366323, 06:12:03 DEBUG "0.0014333333333297563" 06:12:03 DEBUG "556.2154009035116" 06:12:03 DEBUG ] 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 DEBUG ] 06:12:03 DEBUG ] 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:03 DEBUG } 06:12:35 INFO 06:12:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:12:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:12:35 INFO [loop_until]: OK (rc = 0) 06:12:35 DEBUG --- stdout --- 06:12:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 123m 13387Mi rcs-687d95c5b-qs2xv 153m 427Mi rcs-687d95c5b-xxxmk 2m 395Mi 06:12:35 DEBUG --- stderr --- 06:12:35 DEBUG 06:12:37 INFO 06:12:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:12:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:12:37 INFO [loop_until]: OK (rc = 0) 06:12:37 DEBUG --- stdout --- 06:12:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 241m 1% 3431Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 218m 0% 18890Mi 66% 06:12:37 DEBUG --- stderr --- 06:12:37 DEBUG 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 ERROR [http_cmd]: ERROR 06:13:02 ERROR --- status code --- 06:13:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:13:02 ERROR --- http response --- 06:13:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:13:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366382" 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- status code --- 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- status code --- 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG --- http response --- 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 INFO [http_cmd]: http status code OK 06:13:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366383" 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG { 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG "data": { 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG "result": [ 06:13:03 DEBUG { 06:13:03 DEBUG "metric": { 06:13:03 DEBUG "instance": "10.142.0.11:9100" 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG [...] 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG "instance": "10.142.0.51:9100" 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 1735366382, 06:13:03 DEBUG "25122.13333333334" 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG } 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366383" 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366383" 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG { 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG "data": { 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG "result": [ 06:13:03 DEBUG { 06:13:03 DEBUG "metric": { 06:13:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG [...] 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 1735366383, 06:13:03 DEBUG "0.2011016669800718" 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG ] 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366383" 06:13:03 DEBUG } 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366383" 06:13:03 DEBUG } 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366383" 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 DEBUG --- status code --- 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:13:03 DEBUG { 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG { 06:13:03 DEBUG "data": { 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG "data": { 06:13:03 DEBUG "result": [ 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG { 06:13:03 DEBUG "result": [ 06:13:03 DEBUG "metric": { 06:13:03 DEBUG { 06:13:03 DEBUG "pod": "ds-idrepo-0" 06:13:03 DEBUG "metric": { 06:13:03 DEBUG }, 06:13:03 DEBUG "instance": "10.142.0.11:9100" 06:13:03 DEBUG "value": [ 06:13:03 DEBUG }, 06:13:03 DEBUG 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366383" 06:13:03 DEBUG 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 DEBUG 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG [...] 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG [...] 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:13:03 DEBUG 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:13:03 DEBUG 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG 06:13:03 DEBUG { 06:13:03 DEBUG }, 06:13:03 DEBUG { 06:13:03 DEBUG "instance": "10.142.0.51:9100" 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG "value": [ 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG }, 06:13:03 DEBUG "data": { 06:13:03 DEBUG 1735366383, 06:13:03 DEBUG "data": { 06:13:03 DEBUG "value": [ 06:13:03 DEBUG "resultType": "vector", 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 DEBUG "831672320" 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG 1735366383, 06:13:03 DEBUG "result": [ 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG ] 06:13:03 DEBUG "result": [ 06:13:03 DEBUG "0" 06:13:03 DEBUG { 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG } 06:13:03 DEBUG { 06:13:03 DEBUG ] 06:13:03 DEBUG "metric": { 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG ] 06:13:03 DEBUG "metric": { 06:13:03 DEBUG } 06:13:03 DEBUG "instance": "10.142.0.11:9100" 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:13:03 DEBUG } 06:13:03 DEBUG "instance": "10.142.0.11:9100" 06:13:03 DEBUG ] 06:13:03 DEBUG }, 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG } 06:13:03 DEBUG }, 06:13:03 DEBUG } 06:13:03 DEBUG "value": [ 06:13:03 DEBUG { 06:13:03 DEBUG "value": [ 06:13:03 DEBUG } 06:13:03 DEBUG 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG "data": { 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG 06:13:03 DEBUG [...] 06:13:03 DEBUG "result": [ 06:13:03 DEBUG [...] 06:13:03 DEBUG 06:13:03 DEBUG { 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG "metric": { 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:13:03 DEBUG 06:13:03 DEBUG "instance": "10.142.0.51:9100" 06:13:03 DEBUG }, 06:13:03 DEBUG "instance": "10.142.0.51:9100" 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 1735366383, 06:13:03 DEBUG [...] 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG 1735366383, 06:13:03 DEBUG "2405174.433333334" 06:13:03 DEBUG 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG "2422570.4000000004" 06:13:03 DEBUG ] 06:13:03 DEBUG 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:13:03 DEBUG } 06:13:03 DEBUG ] 06:13:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG }, 06:13:03 DEBUG { 06:13:03 DEBUG } 06:13:03 DEBUG } 06:13:03 DEBUG "value": [ 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG } 06:13:03 DEBUG 1735366383, 06:13:03 DEBUG "14896275456" 06:13:03 DEBUG "data": { 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG ] 06:13:03 DEBUG "result": [ 06:13:03 DEBUG } 06:13:03 DEBUG { 06:13:03 DEBUG } 06:13:03 DEBUG "metric": { 06:13:03 DEBUG "pod": "ds-cts-0" 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG [...] 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG "pod": "ds-idrepo-0" 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 1735366383, 06:13:03 DEBUG "0" 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG } 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366383" 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG { 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG "data": { 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG "result": [ 06:13:03 DEBUG { 06:13:03 DEBUG "metric": { 06:13:03 DEBUG "pod": "ds-cts-0" 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG [...] 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG "pod": "ds-idrepo-0" 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366383" 06:13:03 DEBUG 1735366383, 06:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366383" 06:13:03 DEBUG "0.00406220691964357" 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG } 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 INFO [http_cmd]: http status code OK 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG --- status code --- 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG --- http response --- 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG ----- output ----- 06:13:03 DEBUG { 06:13:03 DEBUG { 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG "status": "success", 06:13:03 DEBUG "data": { 06:13:03 DEBUG "data": { 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG "resultType": "vector", 06:13:03 DEBUG "result": [ 06:13:03 DEBUG "result": [ 06:13:03 DEBUG { 06:13:03 DEBUG { 06:13:03 DEBUG "metric": { 06:13:03 DEBUG "metric": { 06:13:03 DEBUG "instance": "10.142.0.11:9100" 06:13:03 DEBUG "pod": "ds-cts-0" 06:13:03 DEBUG }, 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG [...] 06:13:03 DEBUG [...] 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG 06:13:03 DEBUG "instance": "10.142.0.51:9100" 06:13:03 DEBUG "pod": "ds-idrepo-0" 06:13:03 DEBUG }, 06:13:03 DEBUG }, 06:13:03 DEBUG "value": [ 06:13:03 DEBUG "value": [ 06:13:03 DEBUG 1735366383, 06:13:03 DEBUG 1735366383, 06:13:03 DEBUG "0.0006000000000009702" 06:13:03 DEBUG "772.8767368892873" 06:13:03 DEBUG ] 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG } 06:13:03 DEBUG ] 06:13:03 DEBUG ] 06:13:03 DEBUG } 06:13:03 DEBUG } 06:13:03 DEBUG } 06:13:03 DEBUG } 06:13:35 INFO 06:13:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:13:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:13:35 INFO [loop_until]: OK (rc = 0) 06:13:35 DEBUG --- stdout --- 06:13:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 123m 13385Mi rcs-687d95c5b-qs2xv 155m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:13:35 DEBUG --- stderr --- 06:13:35 DEBUG 06:13:37 INFO 06:13:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:13:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:13:37 INFO [loop_until]: OK (rc = 0) 06:13:37 DEBUG --- stdout --- 06:13:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 224m 1% 3445Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 178m 0% 18884Mi 66% 06:13:37 DEBUG --- stderr --- 06:13:37 DEBUG 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 ERROR [http_cmd]: ERROR 06:14:02 ERROR --- status code --- 06:14:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:14:02 ERROR --- http response --- 06:14:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:14:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- http response --- 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366442" 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: http status code OK 06:14:02 DEBUG --- status code --- 06:14:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:02 DEBUG --- http response --- 06:14:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366442" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366443" 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366443" 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG { 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG "data": { 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG "result": [ 06:14:03 DEBUG { 06:14:03 DEBUG "metric": { 06:14:03 DEBUG "instance": "10.142.0.11:9100" 06:14:03 DEBUG }, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG [...] 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG "instance": "10.142.0.51:9100" 06:14:03 DEBUG }, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG "18978.133333333335" 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366443" 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG { 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG "data": { 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG "result": [ 06:14:03 DEBUG { 06:14:03 DEBUG "metric": { 06:14:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:14:03 DEBUG }, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG [...] 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:14:03 DEBUG }, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG "0.10245171499718383" 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366443" 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:14:03 DEBUG { 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG { 06:14:03 DEBUG "data": { 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG "data": { 06:14:03 DEBUG "result": [ 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG { 06:14:03 DEBUG "result": [ 06:14:03 DEBUG "metric": { 06:14:03 DEBUG { 06:14:03 DEBUG "pod": "ds-idrepo-0" 06:14:03 DEBUG "metric": { 06:14:03 DEBUG }, 06:14:03 DEBUG "instance": "10.142.0.11:9100" 06:14:03 DEBUG "value": [ 06:14:03 DEBUG }, 06:14:03 DEBUG 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG [...] 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG [...] 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:14:03 DEBUG 06:14:03 DEBUG }, 06:14:03 DEBUG "instance": "10.142.0.51:9100" 06:14:03 DEBUG "value": [ 06:14:03 DEBUG }, 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG "831672320" 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG ] 06:14:03 DEBUG "0.00033333333333303023" 06:14:03 DEBUG } 06:14:03 DEBUG ] 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG { 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG "data": { 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG "result": [ 06:14:03 DEBUG { 06:14:03 DEBUG "metric": { 06:14:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:14:03 DEBUG }, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG [...] 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:14:03 DEBUG }, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG "14893129728" 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366443" 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG { 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG { 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG "data": { 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366443" 06:14:03 DEBUG "data": { 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366443" 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG "result": [ 06:14:03 DEBUG "result": [ 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:14:03 DEBUG { 06:14:03 DEBUG { 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG "metric": { 06:14:03 DEBUG "instance": "10.142.0.11:9100" 06:14:03 DEBUG { 06:14:03 DEBUG { 06:14:03 DEBUG "metric": { 06:14:03 DEBUG }, 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG "instance": "10.142.0.11:9100" 06:14:03 DEBUG "value": [ 06:14:03 DEBUG "data": { 06:14:03 DEBUG "data": { 06:14:03 DEBUG }, 06:14:03 DEBUG 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 06:14:03 DEBUG "result": [ 06:14:03 DEBUG "result": [ 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG { 06:14:03 DEBUG { 06:14:03 DEBUG 06:14:03 DEBUG [...] 06:14:03 DEBUG "metric": { 06:14:03 DEBUG "metric": { 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG "pod": "ds-idrepo-0" 06:14:03 DEBUG "pod": "ds-idrepo-0" 06:14:03 DEBUG [...] 06:14:03 DEBUG 06:14:03 DEBUG }, 06:14:03 DEBUG }, 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG "value": [ 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 06:14:03 DEBUG "instance": "10.142.0.51:9100" 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG }, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG "2413359.733333334" 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG "instance": "10.142.0.51:9100" 06:14:03 DEBUG ] 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG }, 06:14:03 DEBUG } 06:14:03 DEBUG [...] 06:14:03 DEBUG [...] 06:14:03 DEBUG "value": [ 06:14:03 DEBUG ] 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG } 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG "2430758.0666666673" 06:14:03 DEBUG } 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG ] 06:14:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:14:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:14:03 DEBUG } 06:14:03 DEBUG }, 06:14:03 DEBUG }, 06:14:03 DEBUG ] 06:14:03 DEBUG "value": [ 06:14:03 DEBUG "value": [ 06:14:03 DEBUG } 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG } 06:14:03 DEBUG "0.0013106694654264786" 06:14:03 DEBUG "0" 06:14:03 DEBUG ] 06:14:03 DEBUG ] 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG } 06:14:03 INFO [http_cmd]: http status code OK 06:14:03 DEBUG } 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG ] 06:14:03 DEBUG --- status code --- 06:14:03 DEBUG ] 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG } 06:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:03 DEBUG } 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG } 06:14:03 DEBUG --- http response --- 06:14:03 DEBUG } 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG ----- output ----- 06:14:03 DEBUG { 06:14:03 DEBUG { 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG "status": "success", 06:14:03 DEBUG "data": { 06:14:03 DEBUG "data": { 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG "resultType": "vector", 06:14:03 DEBUG "result": [ 06:14:03 DEBUG "result": [ 06:14:03 DEBUG { 06:14:03 DEBUG { 06:14:03 DEBUG "metric": { 06:14:03 DEBUG "metric": { 06:14:03 DEBUG "pod": "ds-idrepo-0" 06:14:03 DEBUG "instance": "10.142.0.11:9100" 06:14:03 DEBUG }, 06:14:03 DEBUG }, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG [...] 06:14:03 DEBUG [...] 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG 06:14:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:14:03 DEBUG "instance": "10.142.0.51:9100" 06:14:03 DEBUG }, 06:14:03 DEBUG }, 06:14:03 DEBUG "value": [ 06:14:03 DEBUG "value": [ 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG 1735366443, 06:14:03 DEBUG "702.1867206435102" 06:14:03 DEBUG "0.0004333333333306656" 06:14:03 DEBUG ] 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:03 DEBUG ] 06:14:03 DEBUG ] 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:03 DEBUG } 06:14:35 INFO 06:14:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:14:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:14:35 INFO [loop_until]: OK (rc = 0) 06:14:35 DEBUG --- stdout --- 06:14:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 620m 13391Mi rcs-687d95c5b-qs2xv 120m 426Mi rcs-687d95c5b-xxxmk 2m 395Mi 06:14:35 DEBUG --- stderr --- 06:14:35 DEBUG 06:14:37 INFO 06:14:37 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:14:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:14:38 INFO [loop_until]: OK (rc = 0) 06:14:38 DEBUG --- stdout --- 06:14:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 176m 1% 3404Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 686m 2% 18890Mi 66% 06:14:38 DEBUG --- stderr --- 06:14:38 DEBUG 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- http response --- 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG --- status code --- 06:15:02 ERROR [http_cmd]: ERROR 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG --- status code --- 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 ERROR --- status code --- 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- http response --- 06:15:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 ERROR --- http response --- 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366502" 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:02 INFO [http_cmd]: http status code OK 06:15:02 DEBUG --- status code --- 06:15:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:02 DEBUG --- http response --- 06:15:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366503" 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG --- status code --- 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG --- http response --- 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366503" 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG { 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG "data": { 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG "result": [ 06:15:03 DEBUG { 06:15:03 DEBUG "metric": { 06:15:03 DEBUG "instance": "10.142.0.11:9100" 06:15:03 DEBUG }, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG [...] 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG "instance": "10.142.0.51:9100" 06:15:03 DEBUG }, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG "25668.26666666667" 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG { 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG "data": { 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG "result": [ 06:15:03 DEBUG { 06:15:03 DEBUG "metric": { 06:15:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:15:03 DEBUG }, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG [...] 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:15:03 DEBUG }, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG "0.4849242524647779" 06:15:03 DEBUG ] 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366503" 06:15:03 DEBUG } 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366503" 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG { 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG "data": { 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG "result": [ 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:15:03 DEBUG { 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG "metric": { 06:15:03 DEBUG { 06:15:03 DEBUG "pod": "ds-idrepo-0" 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG }, 06:15:03 DEBUG "data": { 06:15:03 DEBUG "value": [ 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG 06:15:03 DEBUG "result": [ 06:15:03 DEBUG 06:15:03 DEBUG { 06:15:03 DEBUG 06:15:03 DEBUG "metric": { 06:15:03 DEBUG [...] 06:15:03 DEBUG "instance": "10.142.0.11:9100" 06:15:03 DEBUG 06:15:03 DEBUG }, 06:15:03 DEBUG 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:15:03 DEBUG 06:15:03 DEBUG }, 06:15:03 DEBUG 06:15:03 DEBUG "value": [ 06:15:03 DEBUG [...] 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG 06:15:03 DEBUG "831672320" 06:15:03 DEBUG 06:15:03 DEBUG ] 06:15:03 DEBUG 06:15:03 DEBUG } 06:15:03 DEBUG "instance": "10.142.0.51:9100" 06:15:03 DEBUG ] 06:15:03 DEBUG }, 06:15:03 DEBUG } 06:15:03 DEBUG "value": [ 06:15:03 DEBUG } 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG "0" 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG { 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG "data": { 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG "result": [ 06:15:03 DEBUG { 06:15:03 DEBUG "metric": { 06:15:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:15:03 DEBUG }, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG [...] 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:15:03 DEBUG }, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG "14894809088" 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366503" 06:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366503" 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG --- http response --- 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG { 06:15:03 DEBUG { 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG "data": { 06:15:03 DEBUG "data": { 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG "result": [ 06:15:03 DEBUG "result": [ 06:15:03 DEBUG { 06:15:03 DEBUG { 06:15:03 DEBUG "metric": { 06:15:03 DEBUG "metric": { 06:15:03 DEBUG "instance": "10.142.0.11:9100" 06:15:03 DEBUG "instance": "10.142.0.11:9100" 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG }, 06:15:03 DEBUG }, 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG "value": [ 06:15:03 DEBUG "value": [ 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG [...] 06:15:03 DEBUG [...] 06:15:03 DEBUG { 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG "data": { 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG "instance": "10.142.0.51:9100" 06:15:03 DEBUG "instance": "10.142.0.51:9100" 06:15:03 DEBUG "result": [ 06:15:03 DEBUG }, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG { 06:15:03 DEBUG }, 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG "metric": { 06:15:03 DEBUG "pod": "ds-idrepo-0" 06:15:03 DEBUG "1632475.5333333337" 06:15:03 DEBUG "value": [ 06:15:03 DEBUG }, 06:15:03 DEBUG ] 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG } 06:15:03 DEBUG "1615101.7000000007" 06:15:03 DEBUG 06:15:03 DEBUG ] 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:03 DEBUG 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG } 06:15:03 DEBUG [...] 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG } 06:15:03 DEBUG 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 INFO [http_cmd]: http status code OK 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG --- status code --- 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:15:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG --- http response --- 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG }, 06:15:03 DEBUG { 06:15:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:15:03 DEBUG { 06:15:03 DEBUG "value": [ 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG ----- output ----- 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG "data": { 06:15:03 DEBUG { 06:15:03 DEBUG "data": { 06:15:03 DEBUG "0.00124748319842365" 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG "status": "success", 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG ] 06:15:03 DEBUG "result": [ 06:15:03 DEBUG "data": { 06:15:03 DEBUG "result": [ 06:15:03 DEBUG } 06:15:03 DEBUG { 06:15:03 DEBUG "resultType": "vector", 06:15:03 DEBUG { 06:15:03 DEBUG ] 06:15:03 DEBUG "metric": { 06:15:03 DEBUG "result": [ 06:15:03 DEBUG "metric": { 06:15:03 DEBUG } 06:15:03 DEBUG "pod": "ds-idrepo-0" 06:15:03 DEBUG { 06:15:03 DEBUG "instance": "10.142.0.11:9100" 06:15:03 DEBUG } 06:15:03 DEBUG }, 06:15:03 DEBUG "metric": { 06:15:03 DEBUG }, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG "pod": "ds-idrepo-0" 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 06:15:03 DEBUG }, 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG [...] 06:15:03 DEBUG 06:15:03 DEBUG [...] 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG [...] 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG 06:15:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:15:03 DEBUG 06:15:03 DEBUG "instance": "10.142.0.51:9100" 06:15:03 DEBUG }, 06:15:03 DEBUG 06:15:03 DEBUG }, 06:15:03 DEBUG "value": [ 06:15:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:15:03 DEBUG "value": [ 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG }, 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG "571.5328376329064" 06:15:03 DEBUG "value": [ 06:15:03 DEBUG "0.0006666666666613236" 06:15:03 DEBUG ] 06:15:03 DEBUG 1735366503, 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG "0" 06:15:03 DEBUG } 06:15:03 DEBUG ] 06:15:03 DEBUG ] 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:03 DEBUG ] 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:03 DEBUG } 06:15:35 INFO 06:15:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:15:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:15:35 INFO [loop_until]: OK (rc = 0) 06:15:35 DEBUG --- stdout --- 06:15:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 60m 13391Mi rcs-687d95c5b-qs2xv 82m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:15:35 DEBUG --- stderr --- 06:15:35 DEBUG 06:15:38 INFO 06:15:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:15:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:15:38 INFO [loop_until]: OK (rc = 0) 06:15:38 DEBUG --- stdout --- 06:15:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 144m 0% 3418Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 119m 0% 18887Mi 66% 06:15:38 DEBUG --- stderr --- 06:15:38 DEBUG 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 ERROR [http_cmd]: ERROR 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG --- status code --- 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 ERROR --- status code --- 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG --- status code --- 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG --- status code --- 06:16:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 ERROR --- http response --- 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366562" 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:02 INFO [http_cmd]: http status code OK 06:16:02 DEBUG --- status code --- 06:16:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:02 DEBUG --- http response --- 06:16:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366563" 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG { 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG "data": { 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG "result": [ 06:16:03 DEBUG { 06:16:03 DEBUG "metric": { 06:16:03 DEBUG "instance": "10.142.0.11:9100" 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG [...] 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG "instance": "10.142.0.51:9100" 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG "20616.533333333336" 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366563" 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366563" 06:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG { 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG "data": { 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG "result": [ 06:16:03 DEBUG { 06:16:03 DEBUG "metric": { 06:16:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG [...] 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG "0.07860922616402537" 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG { 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG "data": { 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG "result": [ 06:16:03 DEBUG { 06:16:03 DEBUG "metric": { 06:16:03 DEBUG "pod": "ds-idrepo-0" 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG }, 06:16:03 DEBUG --- status code --- 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG "value": [ 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:16:03 DEBUG [...] 06:16:03 DEBUG { 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG { 06:16:03 DEBUG 06:16:03 DEBUG "data": { 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG "data": { 06:16:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:16:03 DEBUG "result": [ 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG }, 06:16:03 DEBUG { 06:16:03 DEBUG "result": [ 06:16:03 DEBUG "value": [ 06:16:03 DEBUG "metric": { 06:16:03 DEBUG { 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG "instance": "10.142.0.11:9100" 06:16:03 DEBUG "metric": { 06:16:03 DEBUG "831680512" 06:16:03 DEBUG }, 06:16:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:16:03 DEBUG ] 06:16:03 DEBUG "value": [ 06:16:03 DEBUG }, 06:16:03 DEBUG } 06:16:03 DEBUG 06:16:03 DEBUG "value": [ 06:16:03 DEBUG ] 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG } 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG } 06:16:03 DEBUG [...] 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG [...] 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG "instance": "10.142.0.51:9100" 06:16:03 DEBUG 06:16:03 DEBUG }, 06:16:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:16:03 DEBUG "value": [ 06:16:03 DEBUG }, 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG "0.00033333333333303023" 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG ] 06:16:03 DEBUG "14893584384" 06:16:03 DEBUG } 06:16:03 DEBUG ] 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366563" 06:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366563" 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG { 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG "data": { 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG "result": [ 06:16:03 DEBUG { 06:16:03 DEBUG "metric": { 06:16:03 DEBUG "instance": "10.142.0.11:9100" 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG [...] 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG "instance": "10.142.0.51:9100" 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG "1239630.7333333336" 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG { 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG "data": { 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG "result": [ 06:16:03 DEBUG { 06:16:03 DEBUG "metric": { 06:16:03 DEBUG "instance": "10.142.0.11:9100" 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG [...] 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG "instance": "10.142.0.51:9100" 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG "1254037.466666667" 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG { 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG "data": { 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG "result": [ 06:16:03 DEBUG { 06:16:03 DEBUG { 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG "metric": { 06:16:03 DEBUG "data": { 06:16:03 DEBUG "pod": "ds-idrepo-0" 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG }, 06:16:03 DEBUG "result": [ 06:16:03 DEBUG "value": [ 06:16:03 DEBUG { 06:16:03 DEBUG 06:16:03 DEBUG "metric": { 06:16:03 DEBUG 06:16:03 DEBUG "pod": "ds-idrepo-0" 06:16:03 DEBUG 06:16:03 DEBUG }, 06:16:03 DEBUG [...] 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:16:03 DEBUG [...] 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG }, 06:16:03 INFO [http_cmd]: http status code OK 06:16:03 DEBUG 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG "value": [ 06:16:03 DEBUG --- status code --- 06:16:03 DEBUG 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:03 DEBUG 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG "0.001282518657868513" 06:16:03 DEBUG --- http response --- 06:16:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:16:03 DEBUG ] 06:16:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:16:03 DEBUG }, 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG } 06:16:03 DEBUG ----- output ----- 06:16:03 DEBUG "value": [ 06:16:03 DEBUG { 06:16:03 DEBUG ] 06:16:03 DEBUG { 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG } 06:16:03 DEBUG "status": "success", 06:16:03 DEBUG "759.9483122839665" 06:16:03 DEBUG "data": { 06:16:03 DEBUG } 06:16:03 DEBUG "data": { 06:16:03 DEBUG ] 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG "resultType": "vector", 06:16:03 DEBUG } 06:16:03 DEBUG "result": [ 06:16:03 DEBUG "result": [ 06:16:03 DEBUG ] 06:16:03 DEBUG { 06:16:03 DEBUG { 06:16:03 DEBUG } 06:16:03 DEBUG "metric": { 06:16:03 DEBUG "metric": { 06:16:03 DEBUG } 06:16:03 DEBUG "instance": "10.142.0.11:9100" 06:16:03 DEBUG "pod": "ds-idrepo-0" 06:16:03 DEBUG }, 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG [...] 06:16:03 DEBUG [...] 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG 06:16:03 DEBUG "instance": "10.142.0.51:9100" 06:16:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:16:03 DEBUG }, 06:16:03 DEBUG }, 06:16:03 DEBUG "value": [ 06:16:03 DEBUG "value": [ 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG 1735366563, 06:16:03 DEBUG "0.00033333333333303023" 06:16:03 DEBUG "0" 06:16:03 DEBUG ] 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 DEBUG ] 06:16:03 DEBUG ] 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:03 DEBUG } 06:16:35 INFO 06:16:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:16:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:16:35 INFO [loop_until]: OK (rc = 0) 06:16:35 DEBUG --- stdout --- 06:16:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 43m 13392Mi rcs-687d95c5b-qs2xv 50m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:16:35 DEBUG --- stderr --- 06:16:35 DEBUG 06:16:38 INFO 06:16:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:16:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:16:38 INFO [loop_until]: OK (rc = 0) 06:16:38 DEBUG --- stdout --- 06:16:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 117m 0% 3426Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 84m 0% 18888Mi 66% 06:16:38 DEBUG --- stderr --- 06:16:38 DEBUG 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:02 ERROR [http_cmd]: ERROR 06:17:02 ERROR --- status code --- 06:17:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:17:02 ERROR --- http response --- 06:17:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:17:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366622" 06:17:02 INFO [http_cmd]: http status code OK 06:17:02 DEBUG --- status code --- 06:17:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:02 DEBUG --- http response --- 06:17:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366623" 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG --- status code --- 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG --- status code --- 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366623" 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 DEBUG --- http response --- 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366623" 06:17:03 DEBUG --- http response --- 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366623" 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG { 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:17:03 DEBUG "data": { 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG { 06:17:03 DEBUG "result": [ 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG { 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:17:03 DEBUG "data": { 06:17:03 DEBUG "metric": { 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG "instance": "10.142.0.11:9100" 06:17:03 DEBUG { 06:17:03 DEBUG "result": [ 06:17:03 DEBUG }, 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG { 06:17:03 DEBUG "value": [ 06:17:03 DEBUG "data": { 06:17:03 DEBUG "metric": { 06:17:03 DEBUG 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:17:03 DEBUG 06:17:03 DEBUG "result": [ 06:17:03 DEBUG { 06:17:03 DEBUG 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG }, 06:17:03 DEBUG "metric": { 06:17:03 DEBUG [...] 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG "value": [ 06:17:03 DEBUG "pod": "ds-idrepo-0" 06:17:03 DEBUG 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG 06:17:03 DEBUG }, 06:17:03 DEBUG 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG 06:17:03 DEBUG "value": [ 06:17:03 DEBUG 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG "instance": "10.142.0.51:9100" 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG [...] 06:17:03 DEBUG 06:17:03 DEBUG }, 06:17:03 DEBUG { 06:17:03 DEBUG { 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG "value": [ 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG 06:17:03 DEBUG [...] 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG "data": { 06:17:03 DEBUG "data": { 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG "37137.06666666667" 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:17:03 DEBUG 06:17:03 DEBUG ] 06:17:03 DEBUG "result": [ 06:17:03 DEBUG "result": [ 06:17:03 DEBUG }, 06:17:03 DEBUG 06:17:03 DEBUG } 06:17:03 DEBUG { 06:17:03 DEBUG { 06:17:03 DEBUG "value": [ 06:17:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:17:03 DEBUG ] 06:17:03 DEBUG "metric": { 06:17:03 DEBUG "metric": { 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG }, 06:17:03 DEBUG } 06:17:03 DEBUG "instance": "10.142.0.11:9100" 06:17:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:17:03 DEBUG "0.0689682439950315" 06:17:03 DEBUG "value": [ 06:17:03 DEBUG } 06:17:03 DEBUG }, 06:17:03 DEBUG }, 06:17:03 DEBUG ] 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG "value": [ 06:17:03 DEBUG "value": [ 06:17:03 DEBUG } 06:17:03 DEBUG "831672320" 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG ] 06:17:03 DEBUG ] 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG } 06:17:03 DEBUG ] 06:17:03 DEBUG [...] 06:17:03 DEBUG 06:17:03 DEBUG } 06:17:03 DEBUG [...] 06:17:03 DEBUG 06:17:03 DEBUG } 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG "instance": "10.142.0.51:9100" 06:17:03 DEBUG 06:17:03 DEBUG }, 06:17:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:17:03 DEBUG "value": [ 06:17:03 DEBUG }, 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG "value": [ 06:17:03 DEBUG "0.0010000000000000382" 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG ] 06:17:03 DEBUG "14895550464" 06:17:03 DEBUG } 06:17:03 DEBUG ] 06:17:03 DEBUG ] 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:03 DEBUG ] 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366623" 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG { 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG "data": { 06:17:03 DEBUG { 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG "result": [ 06:17:03 DEBUG "data": { 06:17:03 DEBUG { 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG "metric": { 06:17:03 DEBUG "result": [ 06:17:03 DEBUG "instance": "10.142.0.11:9100" 06:17:03 DEBUG { 06:17:03 DEBUG }, 06:17:03 DEBUG "metric": { 06:17:03 DEBUG "value": [ 06:17:03 DEBUG "instance": "10.142.0.11:9100" 06:17:03 DEBUG 06:17:03 DEBUG }, 06:17:03 DEBUG 06:17:03 DEBUG "value": [ 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG [...] 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG [...] 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG "instance": "10.142.0.51:9100" 06:17:03 DEBUG 06:17:03 DEBUG }, 06:17:03 DEBUG 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366623" 06:17:03 DEBUG "value": [ 06:17:03 DEBUG "instance": "10.142.0.51:9100" 06:17:03 DEBUG 1735366623, 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366623" 06:17:03 DEBUG }, 06:17:03 DEBUG "1233850.1" 06:17:03 DEBUG "value": [ 06:17:03 DEBUG ] 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG } 06:17:03 DEBUG "1250671.3" 06:17:03 DEBUG ] 06:17:03 DEBUG ] 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:03 DEBUG ] 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366623" 06:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366623" 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG --- http response --- 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:17:03 DEBUG { 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG { 06:17:03 DEBUG "data": { 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG "data": { 06:17:03 DEBUG "result": [ 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG { 06:17:03 DEBUG "result": [ 06:17:03 DEBUG "metric": { 06:17:03 DEBUG { 06:17:03 DEBUG "pod": "ds-idrepo-0" 06:17:03 DEBUG "metric": { 06:17:03 DEBUG }, 06:17:03 DEBUG "pod": "ds-idrepo-0" 06:17:03 DEBUG "value": [ 06:17:03 DEBUG }, 06:17:03 DEBUG 06:17:03 DEBUG "value": [ 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG [...] 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG [...] 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:17:03 DEBUG 06:17:03 DEBUG }, 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:17:03 DEBUG "value": [ 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG }, 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG "0.001254247068751338" 06:17:03 DEBUG "value": [ 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG ] 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG } 06:17:03 DEBUG "1076.0506765448563" 06:17:03 DEBUG ] 06:17:03 DEBUG ] 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:03 DEBUG ] 06:17:03 DEBUG } 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG } 06:17:03 DEBUG { 06:17:03 INFO [http_cmd]: http status code OK 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG --- status code --- 06:17:03 DEBUG "data": { 06:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG --- http response --- 06:17:03 DEBUG "result": [ 06:17:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:17:03 DEBUG { 06:17:03 DEBUG ----- output ----- 06:17:03 DEBUG "metric": { 06:17:03 DEBUG { 06:17:03 DEBUG "pod": "ds-idrepo-0" 06:17:03 DEBUG "status": "success", 06:17:03 DEBUG }, 06:17:03 DEBUG "data": { 06:17:03 DEBUG "value": [ 06:17:03 DEBUG "resultType": "vector", 06:17:03 DEBUG 06:17:03 DEBUG "result": [ 06:17:03 DEBUG 06:17:03 DEBUG { 06:17:03 DEBUG 06:17:03 DEBUG "metric": { 06:17:03 DEBUG [...] 06:17:03 DEBUG "instance": "10.142.0.11:9100" 06:17:03 DEBUG 06:17:03 DEBUG }, 06:17:03 DEBUG 06:17:03 DEBUG "value": [ 06:17:03 DEBUG 06:17:03 DEBUG 06:17:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:17:03 DEBUG 06:17:03 DEBUG }, 06:17:03 DEBUG 06:17:03 DEBUG "value": [ 06:17:03 DEBUG [...] 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG 06:17:03 DEBUG "0" 06:17:03 DEBUG 06:17:03 DEBUG ] 06:17:03 DEBUG 06:17:03 DEBUG } 06:17:03 DEBUG "instance": "10.142.0.51:9100" 06:17:03 DEBUG ] 06:17:03 DEBUG }, 06:17:03 DEBUG } 06:17:03 DEBUG "value": [ 06:17:03 DEBUG } 06:17:03 DEBUG 1735366623, 06:17:03 DEBUG "0.0015666666666675159" 06:17:03 DEBUG ] 06:17:03 DEBUG } 06:17:03 DEBUG ] 06:17:03 DEBUG } 06:17:03 DEBUG } 06:17:35 INFO 06:17:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:17:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:17:35 INFO [loop_until]: OK (rc = 0) 06:17:35 DEBUG --- stdout --- 06:17:35 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 52m 13392Mi rcs-687d95c5b-qs2xv 66m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:17:35 DEBUG --- stderr --- 06:17:35 DEBUG 06:17:38 INFO 06:17:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:17:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:17:38 INFO [loop_until]: OK (rc = 0) 06:17:38 DEBUG --- stdout --- 06:17:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 113m 0% 3446Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 84m 0% 18893Mi 66% 06:17:38 DEBUG --- stderr --- 06:17:38 DEBUG 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366682" 06:18:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366682" 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:02 ERROR [http_cmd]: ERROR 06:18:02 ERROR --- status code --- 06:18:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:18:02 ERROR --- http response --- 06:18:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:18:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:18:02 INFO [http_cmd]: http status code OK 06:18:02 DEBUG --- status code --- 06:18:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:02 DEBUG --- http response --- 06:18:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- status code --- 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG --- status code --- 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366683" 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG --- http response --- 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366683" 06:18:03 DEBUG --- http response --- 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366683" 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366683" 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366683" 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 DEBUG --- status code --- 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 DEBUG --- http response --- 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 DEBUG --- http response --- 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:03 DEBUG --- http response --- 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG { 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG "data": { 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG "result": [ 06:18:03 DEBUG { 06:18:03 DEBUG "metric": { 06:18:03 DEBUG "pod": "ds-idrepo-0" 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG }, 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG "value": [ 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG 06:18:03 DEBUG { 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG 06:18:03 DEBUG "data": { 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG [...] 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG "result": [ 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG { 06:18:03 DEBUG { 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:18:03 DEBUG 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG "metric": { 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:18:03 DEBUG "instance": "10.142.0.11:9100" 06:18:03 DEBUG "data": { 06:18:03 DEBUG { 06:18:03 DEBUG }, 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG }, 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG "value": [ 06:18:03 DEBUG { 06:18:03 DEBUG "value": [ 06:18:03 DEBUG "result": [ 06:18:03 DEBUG "data": { 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG 06:18:03 DEBUG { 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG "831672320" 06:18:03 DEBUG "data": { 06:18:03 DEBUG 06:18:03 DEBUG "metric": { 06:18:03 DEBUG "result": [ 06:18:03 DEBUG ] 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG 06:18:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:18:03 DEBUG { 06:18:03 DEBUG } 06:18:03 DEBUG "result": [ 06:18:03 DEBUG [...] 06:18:03 DEBUG }, 06:18:03 DEBUG "metric": { 06:18:03 DEBUG ] 06:18:03 DEBUG { 06:18:03 DEBUG 06:18:03 DEBUG "value": [ 06:18:03 DEBUG "instance": "10.142.0.11:9100" 06:18:03 DEBUG } 06:18:03 DEBUG "metric": { 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG }, 06:18:03 DEBUG } 06:18:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG "value": [ 06:18:03 DEBUG }, 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366683" 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366683" 06:18:03 DEBUG "instance": "10.142.0.51:9100" 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG "value": [ 06:18:03 DEBUG }, 06:18:03 DEBUG [...] 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG "value": [ 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG 06:18:03 DEBUG [...] 06:18:03 DEBUG 06:18:03 DEBUG "27989.33333333334" 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG [...] 06:18:03 DEBUG ] 06:18:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG } 06:18:03 DEBUG }, 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG ] 06:18:03 DEBUG "value": [ 06:18:03 DEBUG "instance": "10.142.0.51:9100" 06:18:03 DEBUG 06:18:03 DEBUG } 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG }, 06:18:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:18:03 DEBUG } 06:18:03 DEBUG "0.03590569165293221" 06:18:03 DEBUG "value": [ 06:18:03 DEBUG }, 06:18:03 DEBUG ] 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG "value": [ 06:18:03 DEBUG } 06:18:03 DEBUG "0.0003333333333339776" 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG "14898147328" 06:18:03 DEBUG ] 06:18:03 DEBUG ] 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG --- http response --- 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366683" 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366683" 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG { 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG "data": { 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG "result": [ 06:18:03 DEBUG { 06:18:03 DEBUG { 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG "data": { 06:18:03 DEBUG "metric": { 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG "instance": "10.142.0.11:9100" 06:18:03 DEBUG "result": [ 06:18:03 DEBUG }, 06:18:03 DEBUG { 06:18:03 DEBUG "value": [ 06:18:03 DEBUG "metric": { 06:18:03 DEBUG 06:18:03 DEBUG "instance": "10.142.0.11:9100" 06:18:03 DEBUG 06:18:03 DEBUG }, 06:18:03 DEBUG 06:18:03 DEBUG "value": [ 06:18:03 DEBUG [...] 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG [...] 06:18:03 DEBUG "instance": "10.142.0.51:9100" 06:18:03 DEBUG 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366683" 06:18:03 DEBUG }, 06:18:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366683" 06:18:03 DEBUG 06:18:03 DEBUG "value": [ 06:18:03 DEBUG 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG "instance": "10.142.0.51:9100" 06:18:03 DEBUG "78648.43333333336" 06:18:03 DEBUG }, 06:18:03 DEBUG ] 06:18:03 DEBUG "value": [ 06:18:03 DEBUG } 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG ] 06:18:03 DEBUG "95626.66666666669" 06:18:03 DEBUG } 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG { 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG "data": { 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG "result": [ 06:18:03 DEBUG { 06:18:03 DEBUG "metric": { 06:18:03 DEBUG "pod": "ds-idrepo-0" 06:18:03 DEBUG }, 06:18:03 DEBUG "value": [ 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG [...] 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:18:03 DEBUG }, 06:18:03 DEBUG "value": [ 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG "782.9160247182368" 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG { 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG "data": { 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG "result": [ 06:18:03 DEBUG { 06:18:03 DEBUG "metric": { 06:18:03 DEBUG "pod": "ds-idrepo-0" 06:18:03 DEBUG }, 06:18:03 DEBUG "value": [ 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG [...] 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:18:03 DEBUG }, 06:18:03 DEBUG "value": [ 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG "0.0013976174020033205" 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 INFO [http_cmd]: http status code OK 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG --- status code --- 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:18:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG --- http response --- 06:18:03 DEBUG { 06:18:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG ----- output ----- 06:18:03 DEBUG "data": { 06:18:03 DEBUG { 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG "status": "success", 06:18:03 DEBUG "result": [ 06:18:03 DEBUG "data": { 06:18:03 DEBUG { 06:18:03 DEBUG "resultType": "vector", 06:18:03 DEBUG "metric": { 06:18:03 DEBUG "result": [ 06:18:03 DEBUG "pod": "ds-idrepo-0" 06:18:03 DEBUG { 06:18:03 DEBUG }, 06:18:03 DEBUG "metric": { 06:18:03 DEBUG "value": [ 06:18:03 DEBUG "instance": "10.142.0.11:9100" 06:18:03 DEBUG 06:18:03 DEBUG }, 06:18:03 DEBUG 06:18:03 DEBUG "value": [ 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG [...] 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG [...] 06:18:03 DEBUG 06:18:03 DEBUG 06:18:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:18:03 DEBUG 06:18:03 DEBUG }, 06:18:03 DEBUG 06:18:03 DEBUG "value": [ 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG "instance": "10.142.0.51:9100" 06:18:03 DEBUG "0" 06:18:03 DEBUG }, 06:18:03 DEBUG ] 06:18:03 DEBUG "value": [ 06:18:03 DEBUG } 06:18:03 DEBUG 1735366683, 06:18:03 DEBUG ] 06:18:03 DEBUG "0.0006666666666679552" 06:18:03 DEBUG } 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:03 DEBUG ] 06:18:03 DEBUG } 06:18:03 DEBUG } 06:18:35 INFO 06:18:35 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:18:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:18:36 INFO [loop_until]: OK (rc = 0) 06:18:36 DEBUG --- stdout --- 06:18:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13392Mi rcs-687d95c5b-qs2xv 1m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:18:36 DEBUG --- stderr --- 06:18:36 DEBUG 06:18:38 INFO 06:18:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:18:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:18:38 INFO [loop_until]: OK (rc = 0) 06:18:38 DEBUG --- stdout --- 06:18:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 125m 0% 3400Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 101m 0% 18891Mi 66% 06:18:38 DEBUG --- stderr --- 06:18:38 DEBUG 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG --- http response --- 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366742" 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG --- http response --- 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG --- status code --- 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 ERROR [http_cmd]: ERROR 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG --- status code --- 06:19:02 ERROR --- status code --- 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG --- status code --- 06:19:02 ERROR --- http response --- 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 DEBUG --- http response --- 06:19:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:02 INFO [http_cmd]: http status code OK 06:19:02 DEBUG --- status code --- 06:19:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:02 DEBUG --- http response --- 06:19:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366743" 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366743" 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- http response --- 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG { 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG "data": { 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG "result": [ 06:19:03 DEBUG { 06:19:03 DEBUG "metric": { 06:19:03 DEBUG "pod": "ds-idrepo-0" 06:19:03 DEBUG }, 06:19:03 DEBUG "value": [ 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG [...] 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:19:03 DEBUG }, 06:19:03 DEBUG "value": [ 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG "831672320" 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG } 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG { 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG "data": { 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG "result": [ 06:19:03 DEBUG { 06:19:03 DEBUG "metric": { 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:19:03 DEBUG "instance": "10.142.0.11:9100" 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG }, 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG { 06:19:03 DEBUG "value": [ 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG "data": { 06:19:03 DEBUG 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366743" 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG "result": [ 06:19:03 DEBUG [...] 06:19:03 DEBUG ----- output ----- 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366743" 06:19:03 DEBUG { 06:19:03 DEBUG { 06:19:03 DEBUG 06:19:03 DEBUG { 06:19:03 DEBUG "status": "success", 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366743" 06:19:03 DEBUG "metric": { 06:19:03 DEBUG 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG "data": { 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366743" 06:19:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:19:03 DEBUG 06:19:03 DEBUG "data": { 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG }, 06:19:03 DEBUG "instance": "10.142.0.51:9100" 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG "result": [ 06:19:03 DEBUG "value": [ 06:19:03 DEBUG }, 06:19:03 DEBUG "result": [ 06:19:03 DEBUG { 06:19:03 DEBUG 06:19:03 DEBUG "value": [ 06:19:03 DEBUG { 06:19:03 DEBUG "metric": { 06:19:03 DEBUG 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG "metric": { 06:19:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:19:03 DEBUG 06:19:03 DEBUG "26760.53333333334" 06:19:03 DEBUG "instance": "10.142.0.11:9100" 06:19:03 DEBUG }, 06:19:03 DEBUG [...] 06:19:03 DEBUG ] 06:19:03 DEBUG }, 06:19:03 DEBUG "value": [ 06:19:03 DEBUG 06:19:03 DEBUG } 06:19:03 DEBUG "value": [ 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG ] 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG } 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG [...] 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG } 06:19:03 DEBUG 06:19:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:19:03 DEBUG 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG [...] 06:19:03 DEBUG }, 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366743" 06:19:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366743" 06:19:03 DEBUG 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG 06:19:03 DEBUG "value": [ 06:19:03 DEBUG 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:19:03 DEBUG 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG 06:19:03 DEBUG "0.08900821085834265" 06:19:03 DEBUG }, 06:19:03 DEBUG { 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:19:03 DEBUG "instance": "10.142.0.51:9100" 06:19:03 DEBUG ] 06:19:03 DEBUG "value": [ 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG }, 06:19:03 DEBUG } 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG "data": { 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG { 06:19:03 DEBUG "value": [ 06:19:03 DEBUG ] 06:19:03 DEBUG "14899142656" 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG } 06:19:03 DEBUG ] 06:19:03 DEBUG "result": [ 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:19:03 DEBUG "data": { 06:19:03 DEBUG "0.00033333333333303023" 06:19:03 DEBUG } 06:19:03 DEBUG } 06:19:03 DEBUG { 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG ] 06:19:03 DEBUG ] 06:19:03 DEBUG "metric": { 06:19:03 DEBUG { 06:19:03 DEBUG { 06:19:03 DEBUG "result": [ 06:19:03 DEBUG } 06:19:03 DEBUG } 06:19:03 DEBUG "instance": "10.142.0.11:9100" 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG "data": { 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG }, 06:19:03 DEBUG "data": { 06:19:03 DEBUG { 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG } 06:19:03 DEBUG "value": [ 06:19:03 DEBUG "resultType": "vector", 06:19:03 INFO [http_cmd]: http status code OK 06:19:03 DEBUG "metric": { 06:19:03 DEBUG "result": [ 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG } 06:19:03 DEBUG 06:19:03 DEBUG "result": [ 06:19:03 DEBUG --- status code --- 06:19:03 DEBUG "instance": "10.142.0.11:9100" 06:19:03 DEBUG { 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG 06:19:03 DEBUG { 06:19:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:03 DEBUG }, 06:19:03 DEBUG "metric": { 06:19:03 DEBUG 06:19:03 DEBUG "metric": { 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:19:03 DEBUG --- http response --- 06:19:03 DEBUG "value": [ 06:19:03 DEBUG "pod": "ds-idrepo-0" 06:19:03 DEBUG [...] 06:19:03 DEBUG "pod": "ds-idrepo-0" 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:19:03 DEBUG 06:19:03 DEBUG }, 06:19:03 DEBUG 06:19:03 DEBUG }, 06:19:03 DEBUG { 06:19:03 DEBUG ----- output ----- 06:19:03 DEBUG 06:19:03 DEBUG "value": [ 06:19:03 DEBUG 06:19:03 DEBUG "value": [ 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG "data": { 06:19:03 DEBUG { 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG 06:19:03 DEBUG "status": "success", 06:19:03 DEBUG 06:19:03 DEBUG "instance": "10.142.0.51:9100" 06:19:03 DEBUG 06:19:03 DEBUG "result": [ 06:19:03 DEBUG [...] 06:19:03 DEBUG "data": { 06:19:03 DEBUG 06:19:03 DEBUG }, 06:19:03 DEBUG 06:19:03 DEBUG { 06:19:03 DEBUG 06:19:03 DEBUG "resultType": "vector", 06:19:03 DEBUG [...] 06:19:03 DEBUG "value": [ 06:19:03 DEBUG [...] 06:19:03 DEBUG "metric": { 06:19:03 DEBUG 06:19:03 DEBUG "result": [ 06:19:03 DEBUG 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG 06:19:03 DEBUG "pod": "ds-idrepo-0" 06:19:03 DEBUG 06:19:03 DEBUG { 06:19:03 DEBUG 06:19:03 DEBUG "92792.60000000003" 06:19:03 DEBUG 06:19:03 DEBUG }, 06:19:03 DEBUG "instance": "10.142.0.51:9100" 06:19:03 DEBUG "metric": { 06:19:03 DEBUG 06:19:03 DEBUG ] 06:19:03 DEBUG 06:19:03 DEBUG "value": [ 06:19:03 DEBUG }, 06:19:03 DEBUG "instance": "10.142.0.11:9100" 06:19:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:19:03 DEBUG } 06:19:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG [...] 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG 06:19:03 DEBUG }, 06:19:03 DEBUG }, 06:19:03 DEBUG ] 06:19:03 DEBUG }, 06:19:03 DEBUG "value": [ 06:19:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:19:03 DEBUG "value": [ 06:19:03 DEBUG "value": [ 06:19:03 DEBUG } 06:19:03 DEBUG "value": [ 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG }, 06:19:03 DEBUG 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG "0.0012747872102357553" 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG "value": [ 06:19:03 DEBUG 06:19:03 DEBUG } 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG "76742.13333333336" 06:19:03 DEBUG ] 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG 06:19:03 DEBUG "901.9277269353547" 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG "0" 06:19:03 DEBUG [...] 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG } 06:19:03 DEBUG ] 06:19:03 DEBUG 06:19:03 DEBUG } 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG ] 06:19:03 DEBUG 06:19:03 DEBUG } 06:19:03 DEBUG } 06:19:03 DEBUG } 06:19:03 DEBUG "instance": "10.142.0.51:9100" 06:19:03 DEBUG } 06:19:03 DEBUG } 06:19:03 DEBUG }, 06:19:03 DEBUG "value": [ 06:19:03 DEBUG 1735366743, 06:19:03 DEBUG "0.0004999999999995454" 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG ] 06:19:03 DEBUG } 06:19:03 DEBUG } 06:19:36 INFO 06:19:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:19:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:19:36 INFO [loop_until]: OK (rc = 0) 06:19:36 DEBUG --- stdout --- 06:19:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13392Mi rcs-687d95c5b-qs2xv 18m 427Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:19:36 DEBUG --- stderr --- 06:19:36 DEBUG 06:19:38 INFO 06:19:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:19:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:19:38 INFO [loop_until]: OK (rc = 0) 06:19:38 DEBUG --- stdout --- 06:19:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 171m 1% 3434Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 622m 1% 18894Mi 66% 06:19:38 DEBUG --- stderr --- 06:19:38 DEBUG 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366802" 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 ERROR [http_cmd]: ERROR 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 ERROR --- status code --- 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 ERROR --- http response --- 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:20:02 INFO [http_cmd]: http status code OK 06:20:02 DEBUG --- status code --- 06:20:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:02 DEBUG --- http response --- 06:20:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366803" 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366803" 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366803" 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366803" 06:20:03 DEBUG --- http response --- 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366803" 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366803" 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG { 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG "data": { 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG "result": [ 06:20:03 DEBUG { 06:20:03 DEBUG "metric": { 06:20:03 DEBUG "pod": "ds-idrepo-0" 06:20:03 DEBUG }, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG [...] 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:20:03 DEBUG }, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG "831672320" 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366803" 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG { 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG "data": { 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG "result": [ 06:20:03 DEBUG { 06:20:03 DEBUG "metric": { 06:20:03 DEBUG "instance": "10.142.0.11:9100" 06:20:03 DEBUG }, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG [...] 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG "instance": "10.142.0.51:9100" 06:20:03 DEBUG }, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG "13926.400000000003" 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366803" 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG { 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG "data": { 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG "result": [ 06:20:03 DEBUG { 06:20:03 DEBUG "metric": { 06:20:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:20:03 DEBUG }, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG [...] 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:20:03 DEBUG }, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG "0.04301882247400042" 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG { 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG "data": { 06:20:03 DEBUG "resultType": "vector", 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366803" 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG "result": [ 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG { 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366803" 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG "metric": { 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:20:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366803" 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG { 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG "data": { 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG "result": [ 06:20:03 DEBUG }, 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366803" 06:20:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366803" 06:20:03 DEBUG { 06:20:03 DEBUG "metric": { 06:20:03 DEBUG "value": [ 06:20:03 DEBUG "instance": "10.142.0.11:9100" 06:20:03 DEBUG 06:20:03 DEBUG }, 06:20:03 DEBUG 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG [...] 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG [...] 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:20:03 DEBUG 06:20:03 DEBUG }, 06:20:03 DEBUG 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG "value": [ 06:20:03 DEBUG "instance": "10.142.0.51:9100" 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG }, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG "14898368512" 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG ] 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG "0.0006666666666660605" 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:20:03 DEBUG } 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG ] 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:20:03 DEBUG { 06:20:03 DEBUG } 06:20:03 DEBUG ] 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:03 DEBUG { 06:20:03 DEBUG "data": { 06:20:03 DEBUG } 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG "result": [ 06:20:03 DEBUG { 06:20:03 DEBUG "metric": { 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG "data": { 06:20:03 DEBUG "instance": "10.142.0.11:9100" 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG }, 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG "result": [ 06:20:03 DEBUG "value": [ 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG { 06:20:03 DEBUG 06:20:03 INFO [http_cmd]: http status code OK 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:20:03 DEBUG "metric": { 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:20:03 DEBUG 06:20:03 DEBUG --- status code --- 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG "pod": "ds-cts-0" 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG 06:20:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:03 DEBUG { 06:20:03 DEBUG { 06:20:03 DEBUG }, 06:20:03 DEBUG { 06:20:03 DEBUG [...] 06:20:03 DEBUG --- http response --- 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG "value": [ 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG 06:20:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:20:03 DEBUG "data": { 06:20:03 DEBUG "data": { 06:20:03 DEBUG 06:20:03 DEBUG "data": { 06:20:03 DEBUG 06:20:03 DEBUG ----- output ----- 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG 06:20:03 DEBUG { 06:20:03 DEBUG "result": [ 06:20:03 DEBUG "result": [ 06:20:03 DEBUG 06:20:03 DEBUG "result": [ 06:20:03 DEBUG "instance": "10.142.0.51:9100" 06:20:03 DEBUG "status": "success", 06:20:03 DEBUG "data": { 06:20:03 DEBUG { 06:20:03 DEBUG [...] 06:20:03 DEBUG { 06:20:03 DEBUG }, 06:20:03 DEBUG "resultType": "vector", 06:20:03 DEBUG { 06:20:03 DEBUG "metric": { 06:20:03 DEBUG 06:20:03 DEBUG "metric": { 06:20:03 DEBUG "value": [ 06:20:03 DEBUG "result": [ 06:20:03 DEBUG { 06:20:03 DEBUG "metric": { 06:20:03 DEBUG "instance": "10.142.0.11:9100" 06:20:03 DEBUG }, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG [...] 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG "metric": { 06:20:03 DEBUG "pod": "ds-cts-0" 06:20:03 DEBUG 06:20:03 DEBUG "pod": "ds-cts-0" 06:20:03 DEBUG 06:20:03 DEBUG "719833.2333333333" 06:20:03 DEBUG "instance": "10.142.0.11:9100" 06:20:03 DEBUG }, 06:20:03 DEBUG 06:20:03 DEBUG }, 06:20:03 DEBUG 06:20:03 DEBUG ] 06:20:03 DEBUG }, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG "pod": "ds-idrepo-0" 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 06:20:03 DEBUG } 06:20:03 DEBUG "value": [ 06:20:03 DEBUG 06:20:03 DEBUG }, 06:20:03 DEBUG 06:20:03 DEBUG "instance": "10.142.0.51:9100" 06:20:03 DEBUG ] 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG }, 06:20:03 DEBUG } 06:20:03 DEBUG 06:20:03 DEBUG "value": [ 06:20:03 DEBUG [...] 06:20:03 DEBUG 06:20:03 DEBUG "value": [ 06:20:03 DEBUG } 06:20:03 DEBUG 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG 06:20:03 DEBUG [...] 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG [...] 06:20:03 DEBUG "0.015951414560375334" 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG "0.000299999999992906" 06:20:03 DEBUG 06:20:03 DEBUG ] 06:20:03 DEBUG 06:20:03 DEBUG 06:20:03 DEBUG ] 06:20:03 DEBUG 06:20:03 DEBUG } 06:20:03 DEBUG "pod": "ds-idrepo-0" 06:20:03 DEBUG 06:20:03 DEBUG } 06:20:03 DEBUG 06:20:03 DEBUG ] 06:20:03 DEBUG }, 06:20:03 DEBUG "pod": "ds-idrepo-0" 06:20:03 DEBUG ] 06:20:03 DEBUG "instance": "10.142.0.51:9100" 06:20:03 DEBUG } 06:20:03 DEBUG "value": [ 06:20:03 DEBUG }, 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG }, 06:20:03 DEBUG "1236.0618634477555" 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG "value": [ 06:20:03 DEBUG ] 06:20:03 DEBUG "305770.8034703885" 06:20:03 DEBUG 1735366803, 06:20:03 DEBUG } 06:20:03 DEBUG ] 06:20:03 DEBUG "701854.9666666669" 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:03 DEBUG ] 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:03 DEBUG } 06:20:36 INFO 06:20:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:20:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:20:36 INFO [loop_until]: OK (rc = 0) 06:20:36 DEBUG --- stdout --- 06:20:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 66m 13402Mi rcs-687d95c5b-qs2xv 73m 426Mi rcs-687d95c5b-xxxmk 1m 395Mi 06:20:36 DEBUG --- stderr --- 06:20:36 DEBUG 06:20:38 INFO 06:20:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:20:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:20:38 INFO [loop_until]: OK (rc = 0) 06:20:38 DEBUG --- stdout --- 06:20:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 188m 1% 3443Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 158m 0% 18903Mi 66% 06:20:38 DEBUG --- stderr --- 06:20:38 DEBUG 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366862" 06:21:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366862" 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:02 ERROR [http_cmd]: ERROR 06:21:02 ERROR --- status code --- 06:21:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:21:02 ERROR --- http response --- 06:21:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:21:02 INFO [http_cmd]: http status code OK 06:21:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:21:02 DEBUG --- status code --- 06:21:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:02 DEBUG --- http response --- 06:21:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366863" 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366863" 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG { 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG "data": { 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG "result": [ 06:21:03 DEBUG { 06:21:03 DEBUG "metric": { 06:21:03 DEBUG "pod": "ds-idrepo-0" 06:21:03 DEBUG }, 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG [...] 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:21:03 DEBUG }, 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG "832344064" 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366863" 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG { 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG "data": { 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG "result": [ 06:21:03 DEBUG { 06:21:03 DEBUG "metric": { 06:21:03 DEBUG "instance": "10.142.0.11:9100" 06:21:03 DEBUG }, 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG [...] 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG "instance": "10.142.0.51:9100" 06:21:03 DEBUG }, 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG "20206.933333333334" 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG { 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG "data": { 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG "result": [ 06:21:03 DEBUG { 06:21:03 DEBUG "metric": { 06:21:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:21:03 DEBUG }, 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG [...] 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:21:03 DEBUG }, 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG "0.12673526718730463" 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366863" 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG "data": { 06:21:03 DEBUG { 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG "result": [ 06:21:03 DEBUG "data": { 06:21:03 DEBUG { 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG "metric": { 06:21:03 DEBUG "result": [ 06:21:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:21:03 DEBUG { 06:21:03 DEBUG }, 06:21:03 DEBUG "metric": { 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366863" 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG "instance": "10.142.0.11:9100" 06:21:03 DEBUG 06:21:03 DEBUG }, 06:21:03 DEBUG [...] 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:21:03 DEBUG [...] 06:21:03 DEBUG }, 06:21:03 DEBUG 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG 06:21:03 DEBUG "14912196608" 06:21:03 DEBUG "instance": "10.142.0.51:9100" 06:21:03 DEBUG ] 06:21:03 DEBUG }, 06:21:03 DEBUG } 06:21:03 DEBUG "value": [ 06:21:03 DEBUG ] 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG } 06:21:03 DEBUG "0.0003333333333335039" 06:21:03 DEBUG } 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366863" 06:21:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366863" 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG { 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG "data": { 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG "result": [ 06:21:03 DEBUG { 06:21:03 DEBUG "metric": { 06:21:03 DEBUG "instance": "10.142.0.11:9100" 06:21:03 DEBUG }, 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG [...] 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG "instance": "10.142.0.51:9100" 06:21:03 DEBUG }, 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG "1254008.9000000001" 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG { 06:21:03 DEBUG "status": "success", 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG "data": { 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG "result": [ 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG { 06:21:03 INFO [http_cmd]: http status code OK 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:21:03 DEBUG "metric": { 06:21:03 DEBUG "instance": "10.142.0.11:9100" 06:21:03 DEBUG }, 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG --- status code --- 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:03 DEBUG { 06:21:03 DEBUG { 06:21:03 DEBUG [...] 06:21:03 DEBUG { 06:21:03 DEBUG --- http response --- 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:21:03 DEBUG "data": { 06:21:03 DEBUG "data": { 06:21:03 DEBUG 06:21:03 DEBUG "data": { 06:21:03 DEBUG ----- output ----- 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG { 06:21:03 DEBUG "result": [ 06:21:03 DEBUG "result": [ 06:21:03 DEBUG "instance": "10.142.0.51:9100" 06:21:03 DEBUG "result": [ 06:21:03 DEBUG "status": "success", 06:21:03 DEBUG { 06:21:03 DEBUG { 06:21:03 DEBUG }, 06:21:03 DEBUG { 06:21:03 DEBUG "data": { 06:21:03 DEBUG "metric": { 06:21:03 DEBUG "metric": { 06:21:03 DEBUG "value": [ 06:21:03 DEBUG "metric": { 06:21:03 DEBUG "resultType": "vector", 06:21:03 DEBUG "pod": "ds-idrepo-0" 06:21:03 DEBUG "pod": "ds-idrepo-0" 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG "pod": "ds-idrepo-0" 06:21:03 DEBUG "result": [ 06:21:03 DEBUG }, 06:21:03 DEBUG }, 06:21:03 DEBUG "0.00033333333333303023" 06:21:03 DEBUG }, 06:21:03 DEBUG { 06:21:03 DEBUG "value": [ 06:21:03 DEBUG "value": [ 06:21:03 DEBUG ] 06:21:03 DEBUG "value": [ 06:21:03 DEBUG "metric": { 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG } 06:21:03 DEBUG 06:21:03 DEBUG "instance": "10.142.0.11:9100" 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG ] 06:21:03 DEBUG 06:21:03 DEBUG }, 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG } 06:21:03 DEBUG 06:21:03 DEBUG "value": [ 06:21:03 DEBUG [...] 06:21:03 DEBUG [...] 06:21:03 DEBUG } 06:21:03 DEBUG [...] 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG 06:21:03 DEBUG [...] 06:21:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:21:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:21:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:21:03 DEBUG 06:21:03 DEBUG }, 06:21:03 DEBUG }, 06:21:03 DEBUG }, 06:21:03 DEBUG 06:21:03 DEBUG "value": [ 06:21:03 DEBUG "value": [ 06:21:03 DEBUG "value": [ 06:21:03 DEBUG 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG "instance": "10.142.0.51:9100" 06:21:03 DEBUG "0.006933825760764702" 06:21:03 DEBUG "0" 06:21:03 DEBUG "980.8910344414455" 06:21:03 DEBUG }, 06:21:03 DEBUG ] 06:21:03 DEBUG ] 06:21:03 DEBUG ] 06:21:03 DEBUG "value": [ 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 DEBUG 1735366863, 06:21:03 DEBUG ] 06:21:03 DEBUG ] 06:21:03 DEBUG ] 06:21:03 DEBUG "1240139.1666666667" 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:03 DEBUG ] 06:21:03 DEBUG } 06:21:03 DEBUG } 06:21:36 INFO 06:21:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:21:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:21:36 INFO [loop_until]: OK (rc = 0) 06:21:36 DEBUG --- stdout --- 06:21:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 72m 13404Mi rcs-687d95c5b-qs2xv 87m 427Mi rcs-687d95c5b-xxxmk 3m 396Mi 06:21:36 DEBUG --- stderr --- 06:21:36 DEBUG 06:21:38 INFO 06:21:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:21:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:21:38 INFO [loop_until]: OK (rc = 0) 06:21:38 DEBUG --- stdout --- 06:21:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 234m 1% 3439Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 208m 0% 18900Mi 66% 06:21:38 DEBUG --- stderr --- 06:21:38 DEBUG 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: http status code OK 06:22:02 INFO [http_cmd]: http status code OK 06:22:02 INFO [http_cmd]: http status code OK 06:22:02 DEBUG --- status code --- 06:22:02 INFO [http_cmd]: http status code OK 06:22:02 INFO [http_cmd]: http status code OK 06:22:02 DEBUG --- status code --- 06:22:02 DEBUG --- status code --- 06:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:02 DEBUG --- status code --- 06:22:02 DEBUG --- status code --- 06:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:02 DEBUG --- http response --- 06:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:02 DEBUG --- http response --- 06:22:02 DEBUG --- http response --- 06:22:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:02 DEBUG --- http response --- 06:22:02 DEBUG --- http response --- 06:22:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366922" 06:22:02 INFO [http_cmd]: http status code OK 06:22:02 DEBUG --- status code --- 06:22:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:02 DEBUG --- http response --- 06:22:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:02 INFO [http_cmd]: http status code OK 06:22:02 DEBUG --- status code --- 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 ERROR [http_cmd]: ERROR 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 ERROR --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 ERROR --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:22:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366923" 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG --- status code --- 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366923" 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366923" 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 DEBUG --- http response --- 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366923" 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366923" 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG { 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG "data": { 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG "result": [ 06:22:03 DEBUG { 06:22:03 DEBUG "metric": { 06:22:03 DEBUG "pod": "ds-idrepo-0" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG [...] 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG "832434176" 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366923" 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG { 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG "data": { 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG "result": [ 06:22:03 DEBUG { 06:22:03 DEBUG "metric": { 06:22:03 DEBUG "instance": "10.142.0.11:9100" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG [...] 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "instance": "10.142.0.51:9100" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG "31402.66666666667" 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG { 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG "data": { 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG "result": [ 06:22:03 DEBUG { 06:22:03 DEBUG "metric": { 06:22:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG [...] 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG "0.1758926532682099" 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366923" 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG { 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG "data": { 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG "result": [ 06:22:03 DEBUG { 06:22:03 DEBUG "metric": { 06:22:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG [...] 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG "14910382080" 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG { 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG "data": { 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG "result": [ 06:22:03 DEBUG { 06:22:03 DEBUG "metric": { 06:22:03 DEBUG "instance": "10.142.0.11:9100" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG [...] 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "instance": "10.142.0.51:9100" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG "0.0006666666666665342" 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366923" 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG { 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG "data": { 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG "result": [ 06:22:03 DEBUG { 06:22:03 DEBUG "metric": { 06:22:03 DEBUG "instance": "10.142.0.11:9100" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG [...] 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "instance": "10.142.0.51:9100" 06:22:03 DEBUG }, 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG "2167622.6666666674" 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366923" 06:22:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366923" 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG { 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG "data": { 06:22:03 DEBUG "resultType": "vector", 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG "result": [ 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG { 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG "metric": { 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG "pod": "ds-idrepo-0" 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:22:03 DEBUG }, 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG "value": [ 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG { 06:22:03 DEBUG 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG "data": { 06:22:03 DEBUG 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 INFO [http_cmd]: http status code OK 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG [...] 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG --- status code --- 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG "result": [ 06:22:03 DEBUG 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:03 DEBUG { 06:22:03 DEBUG { 06:22:03 DEBUG 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG --- http response --- 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG "metric": { 06:22:03 DEBUG 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:22:03 DEBUG "data": { 06:22:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:22:03 DEBUG "instance": "10.142.0.11:9100" 06:22:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG ----- output ----- 06:22:03 DEBUG }, 06:22:03 DEBUG }, 06:22:03 DEBUG { 06:22:03 DEBUG "result": [ 06:22:03 DEBUG { 06:22:03 DEBUG "value": [ 06:22:03 DEBUG "value": [ 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG { 06:22:03 DEBUG "status": "success", 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG [...] 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "metric": { 06:22:03 DEBUG "data": { 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG "data": { 06:22:03 DEBUG "instance": "10.142.0.51:9100" 06:22:03 DEBUG "pod": "ds-idrepo-0" 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG "0.001342105885121799" 06:22:03 DEBUG "resultType": "vector", 06:22:03 DEBUG }, 06:22:03 DEBUG }, 06:22:03 DEBUG "result": [ 06:22:03 DEBUG ] 06:22:03 DEBUG "result": [ 06:22:03 DEBUG "value": [ 06:22:03 DEBUG "value": [ 06:22:03 DEBUG { 06:22:03 DEBUG } 06:22:03 DEBUG { 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG 06:22:03 DEBUG "metric": { 06:22:03 DEBUG ] 06:22:03 DEBUG "metric": { 06:22:03 DEBUG "0.0011333333333368501" 06:22:03 DEBUG 06:22:03 DEBUG "instance": "10.142.0.11:9100" 06:22:03 DEBUG } 06:22:03 DEBUG "pod": "ds-idrepo-0" 06:22:03 DEBUG ] 06:22:03 DEBUG 06:22:03 DEBUG }, 06:22:03 DEBUG } 06:22:03 DEBUG }, 06:22:03 DEBUG } 06:22:03 DEBUG [...] 06:22:03 DEBUG "value": [ 06:22:03 DEBUG "value": [ 06:22:03 DEBUG ] 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG } 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG } 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:22:03 DEBUG [...] 06:22:03 DEBUG [...] 06:22:03 DEBUG }, 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "value": [ 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG 06:22:03 DEBUG 06:22:03 DEBUG "1323.878592348518" 06:22:03 DEBUG "instance": "10.142.0.51:9100" 06:22:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:22:03 DEBUG ] 06:22:03 DEBUG }, 06:22:03 DEBUG }, 06:22:03 DEBUG } 06:22:03 DEBUG "value": [ 06:22:03 DEBUG "value": [ 06:22:03 DEBUG ] 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG 1735366923, 06:22:03 DEBUG } 06:22:03 DEBUG "2150439.3333333335" 06:22:03 DEBUG "0" 06:22:03 DEBUG } 06:22:03 DEBUG ] 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:03 DEBUG ] 06:22:03 DEBUG ] 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:03 DEBUG } 06:22:36 INFO 06:22:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:22:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:22:36 INFO [loop_until]: OK (rc = 0) 06:22:36 DEBUG --- stdout --- 06:22:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 191m 13405Mi rcs-687d95c5b-qs2xv 145m 427Mi rcs-687d95c5b-xxxmk 1m 396Mi 06:22:36 DEBUG --- stderr --- 06:22:36 DEBUG 06:22:38 INFO 06:22:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:22:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:22:38 INFO [loop_until]: OK (rc = 0) 06:22:38 DEBUG --- stdout --- 06:22:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 105m 0% 3462Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 97m 0% 18906Mi 66% 06:22:38 DEBUG --- stderr --- 06:22:38 DEBUG 06:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735366982" 06:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366982" 06:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366982" 06:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366982" 06:23:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366982" 06:23:02 INFO [http_cmd]: http status code OK 06:23:02 INFO [http_cmd]: http status code OK 06:23:02 INFO [http_cmd]: http status code OK 06:23:02 DEBUG --- status code --- 06:23:02 INFO [http_cmd]: http status code OK 06:23:02 INFO [http_cmd]: http status code OK 06:23:02 DEBUG --- status code --- 06:23:02 DEBUG --- status code --- 06:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:02 DEBUG --- status code --- 06:23:02 DEBUG --- status code --- 06:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:02 DEBUG --- http response --- 06:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:02 DEBUG --- http response --- 06:23:02 DEBUG --- http response --- 06:23:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:02 DEBUG --- http response --- 06:23:02 DEBUG --- http response --- 06:23:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 ERROR [http_cmd]: ERROR 06:23:03 DEBUG --- status code --- 06:23:03 ERROR --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 ERROR --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:23:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735366983" 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735366983" 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366983" 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG --- http response --- 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- http response --- 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735366983" 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG { 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:23:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG "data": { 06:23:03 DEBUG { 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG "result": [ 06:23:03 DEBUG "data": { 06:23:03 DEBUG { 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG "metric": { 06:23:03 DEBUG "result": [ 06:23:03 DEBUG "pod": "ds-idrepo-0" 06:23:03 DEBUG { 06:23:03 DEBUG }, 06:23:03 DEBUG "metric": { 06:23:03 DEBUG "value": [ 06:23:03 DEBUG "instance": "10.142.0.11:9100" 06:23:03 DEBUG 06:23:03 DEBUG }, 06:23:03 DEBUG 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG [...] 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG [...] 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:23:03 DEBUG 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG }, 06:23:03 DEBUG 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG "value": [ 06:23:03 DEBUG "instance": "10.142.0.51:9100" 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG }, 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG "832253952" 06:23:03 DEBUG "value": [ 06:23:03 DEBUG { 06:23:03 DEBUG ] 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG } 06:23:03 DEBUG "18705.06666666667" 06:23:03 DEBUG "data": { 06:23:03 DEBUG ] 06:23:03 DEBUG ] 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG } 06:23:03 DEBUG } 06:23:03 DEBUG "result": [ 06:23:03 DEBUG } 06:23:03 DEBUG ] 06:23:03 DEBUG { 06:23:03 DEBUG } 06:23:03 DEBUG "metric": { 06:23:03 DEBUG } 06:23:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG [...] 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG "0.2176292447679157" 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG } 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366983" 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG { 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG "data": { 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG "result": [ 06:23:03 DEBUG { 06:23:03 DEBUG "metric": { 06:23:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG [...] 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG "14907449344" 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG } 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366983" 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG { 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG "data": { 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG "result": [ 06:23:03 DEBUG { 06:23:03 DEBUG "metric": { 06:23:03 DEBUG "instance": "10.142.0.11:9100" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG [...] 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG "instance": "10.142.0.51:9100" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG "0" 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG } 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG { 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG "data": { 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG "result": [ 06:23:03 DEBUG { 06:23:03 DEBUG "metric": { 06:23:03 DEBUG "instance": "10.142.0.11:9100" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG [...] 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG "instance": "10.142.0.51:9100" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG "2433795.133333334" 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG } 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735366983" 06:23:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735366983" 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG { 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:23:03 DEBUG "data": { 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG { 06:23:03 DEBUG "result": [ 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG { 06:23:03 DEBUG "data": { 06:23:03 DEBUG "metric": { 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG "pod": "ds-cts-0" 06:23:03 DEBUG "result": [ 06:23:03 DEBUG }, 06:23:03 DEBUG { 06:23:03 DEBUG "value": [ 06:23:03 DEBUG "metric": { 06:23:03 DEBUG 06:23:03 DEBUG "instance": "10.142.0.11:9100" 06:23:03 DEBUG 06:23:03 DEBUG }, 06:23:03 DEBUG 06:23:03 DEBUG "value": [ 06:23:03 DEBUG [...] 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG [...] 06:23:03 DEBUG "pod": "ds-idrepo-0" 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:03 DEBUG 06:23:03 DEBUG }, 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG 06:23:03 DEBUG "value": [ 06:23:03 DEBUG { 06:23:03 DEBUG 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG "instance": "10.142.0.51:9100" 06:23:03 DEBUG "0" 06:23:03 DEBUG "data": { 06:23:03 DEBUG }, 06:23:03 DEBUG ] 06:23:03 DEBUG "resultType": "vector", 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG "value": [ 06:23:03 DEBUG } 06:23:03 DEBUG "result": [ 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG ] 06:23:03 DEBUG { 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG "0.00033333333333303023" 06:23:03 DEBUG } 06:23:03 DEBUG "metric": { 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG "pod": "ds-cts-0" 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:03 DEBUG } 06:23:03 DEBUG }, 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG ] 06:23:03 DEBUG "value": [ 06:23:03 DEBUG { 06:23:03 DEBUG } 06:23:03 DEBUG 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG } 06:23:03 DEBUG 06:23:03 DEBUG "data": { 06:23:03 DEBUG 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG [...] 06:23:03 DEBUG "result": [ 06:23:03 DEBUG 06:23:03 DEBUG { 06:23:03 DEBUG 06:23:03 DEBUG "metric": { 06:23:03 DEBUG 06:23:03 DEBUG "pod": "ds-cts-0" 06:23:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:23:03 DEBUG }, 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG 06:23:03 DEBUG "853.1681944268756" 06:23:03 DEBUG 06:23:03 DEBUG ] 06:23:03 DEBUG [...] 06:23:03 DEBUG } 06:23:03 DEBUG 06:23:03 DEBUG ] 06:23:03 DEBUG 06:23:03 DEBUG } 06:23:03 DEBUG 06:23:03 DEBUG } 06:23:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG "0" 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG } 06:23:03 INFO [http_cmd]: http status code OK 06:23:03 DEBUG --- status code --- 06:23:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:03 DEBUG --- http response --- 06:23:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:23:03 DEBUG ----- output ----- 06:23:03 DEBUG { 06:23:03 DEBUG "status": "success", 06:23:03 DEBUG "data": { 06:23:03 DEBUG "resultType": "vector", 06:23:03 DEBUG "result": [ 06:23:03 DEBUG { 06:23:03 DEBUG "metric": { 06:23:03 DEBUG "instance": "10.142.0.11:9100" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG [...] 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG 06:23:03 DEBUG "instance": "10.142.0.51:9100" 06:23:03 DEBUG }, 06:23:03 DEBUG "value": [ 06:23:03 DEBUG 1735366983, 06:23:03 DEBUG "2416565.9000000004" 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG ] 06:23:03 DEBUG } 06:23:03 DEBUG } 06:23:36 INFO 06:23:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:23:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:23:36 INFO [loop_until]: OK (rc = 0) 06:23:36 DEBUG --- stdout --- 06:23:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 119m 13403Mi rcs-687d95c5b-qs2xv 173m 427Mi rcs-687d95c5b-xxxmk 1m 396Mi 06:23:36 DEBUG --- stderr --- 06:23:36 DEBUG 06:23:38 INFO 06:23:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:23:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:23:38 INFO [loop_until]: OK (rc = 0) 06:23:38 DEBUG --- stdout --- 06:23:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 123m 0% 3465Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 108m 0% 18906Mi 66% 06:23:38 DEBUG --- stderr --- 06:23:38 DEBUG 06:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367042" 06:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367042" 06:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367042" 06:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735367042" 06:24:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367042" 06:24:02 INFO [http_cmd]: http status code OK 06:24:02 DEBUG --- status code --- 06:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:02 DEBUG --- http response --- 06:24:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:02 INFO [http_cmd]: http status code OK 06:24:02 DEBUG --- status code --- 06:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:02 DEBUG --- http response --- 06:24:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:02 INFO [http_cmd]: http status code OK 06:24:02 DEBUG --- status code --- 06:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:02 DEBUG --- http response --- 06:24:02 INFO [http_cmd]: http status code OK 06:24:02 INFO [http_cmd]: http status code OK 06:24:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:02 DEBUG --- status code --- 06:24:02 DEBUG --- status code --- 06:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:02 DEBUG --- http response --- 06:24:02 DEBUG --- http response --- 06:24:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735367043" 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- http response --- 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 ERROR [http_cmd]: ERROR 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 ERROR --- status code --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- http response --- 06:24:03 ERROR --- http response --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735367043" 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367043" 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- status code --- 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735367043" 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367043" 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG ----- output ----- 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG { 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG "data": { 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:24:03 DEBUG "result": [ 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG { 06:24:03 DEBUG { 06:24:03 DEBUG { 06:24:03 DEBUG "metric": { 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG "pod": "ds-idrepo-0" 06:24:03 DEBUG "data": { 06:24:03 DEBUG "data": { 06:24:03 DEBUG }, 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG "value": [ 06:24:03 DEBUG "result": [ 06:24:03 DEBUG "result": [ 06:24:03 DEBUG 06:24:03 DEBUG { 06:24:03 DEBUG { 06:24:03 DEBUG 06:24:03 DEBUG "metric": { 06:24:03 DEBUG "metric": { 06:24:03 DEBUG 06:24:03 DEBUG "instance": "10.142.0.11:9100" 06:24:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG [...] 06:24:03 DEBUG }, 06:24:03 DEBUG }, 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG 06:24:03 DEBUG "value": [ 06:24:03 DEBUG "value": [ 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG }, 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG [...] 06:24:03 DEBUG [...] 06:24:03 DEBUG "value": [ 06:24:03 DEBUG { 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG "832253952" 06:24:03 DEBUG "data": { 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG ] 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG "instance": "10.142.0.51:9100" 06:24:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:24:03 DEBUG } 06:24:03 DEBUG "result": [ 06:24:03 DEBUG }, 06:24:03 DEBUG }, 06:24:03 DEBUG ] 06:24:03 DEBUG { 06:24:03 DEBUG "value": [ 06:24:03 DEBUG "value": [ 06:24:03 DEBUG } 06:24:03 DEBUG "metric": { 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG } 06:24:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:24:03 DEBUG "20343.46666666667" 06:24:03 DEBUG "0.15096524566666378" 06:24:03 DEBUG }, 06:24:03 DEBUG ] 06:24:03 DEBUG ] 06:24:03 DEBUG "value": [ 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:03 DEBUG 06:24:03 DEBUG ] 06:24:03 DEBUG ] 06:24:03 DEBUG 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:03 DEBUG 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:03 DEBUG [...] 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:24:03 DEBUG }, 06:24:03 DEBUG "value": [ 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367043" 06:24:03 DEBUG 1735367043, 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG "14903795712" 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG ] 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG } 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG ] 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:24:03 DEBUG } 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG } 06:24:03 DEBUG { 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG "data": { 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG "result": [ 06:24:03 DEBUG { 06:24:03 DEBUG "metric": { 06:24:03 DEBUG "instance": "10.142.0.11:9100" 06:24:03 DEBUG }, 06:24:03 DEBUG "value": [ 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG [...] 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG "instance": "10.142.0.51:9100" 06:24:03 DEBUG }, 06:24:03 DEBUG "value": [ 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG "0.00033333333333303023" 06:24:03 DEBUG ] 06:24:03 DEBUG } 06:24:03 DEBUG ] 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG { 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG "data": { 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG "result": [ 06:24:03 DEBUG { 06:24:03 DEBUG "metric": { 06:24:03 DEBUG "instance": "10.142.0.11:9100" 06:24:03 DEBUG }, 06:24:03 DEBUG "value": [ 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG [...] 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG "instance": "10.142.0.51:9100" 06:24:03 DEBUG }, 06:24:03 DEBUG "value": [ 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG "2419607.400000001" 06:24:03 DEBUG ] 06:24:03 DEBUG } 06:24:03 DEBUG ] 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367043" 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG { 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG "data": { 06:24:03 DEBUG { 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG "result": [ 06:24:03 DEBUG "data": { 06:24:03 DEBUG { 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG "metric": { 06:24:03 DEBUG "result": [ 06:24:03 DEBUG "pod": "ds-idrepo-0" 06:24:03 DEBUG { 06:24:03 DEBUG }, 06:24:03 DEBUG "metric": { 06:24:03 DEBUG "value": [ 06:24:03 DEBUG "instance": "10.142.0.11:9100" 06:24:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367043" 06:24:03 DEBUG 06:24:03 DEBUG }, 06:24:03 DEBUG 06:24:03 DEBUG "value": [ 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG [...] 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG [...] 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:24:03 DEBUG 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG }, 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG "value": [ 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG "instance": "10.142.0.51:9100" 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG "0.0012808344334914476" 06:24:03 DEBUG }, 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG ] 06:24:03 DEBUG "value": [ 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG } 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:24:03 DEBUG ] 06:24:03 DEBUG "0.00043333333333445514" 06:24:03 DEBUG { 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG } 06:24:03 DEBUG ] 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG { 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:03 DEBUG "data": { 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG ] 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG "data": { 06:24:03 DEBUG } 06:24:03 DEBUG "result": [ 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG } 06:24:03 DEBUG { 06:24:03 DEBUG "result": [ 06:24:03 DEBUG "metric": { 06:24:03 DEBUG { 06:24:03 DEBUG "pod": "ds-idrepo-0" 06:24:03 DEBUG "metric": { 06:24:03 DEBUG }, 06:24:03 DEBUG "pod": "ds-idrepo-0" 06:24:03 DEBUG "value": [ 06:24:03 DEBUG }, 06:24:03 DEBUG 06:24:03 DEBUG "value": [ 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG [...] 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG [...] 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:24:03 DEBUG 06:24:03 DEBUG }, 06:24:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:24:03 DEBUG "value": [ 06:24:03 DEBUG }, 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG "value": [ 06:24:03 DEBUG "1002.6960300880711" 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG ] 06:24:03 DEBUG "0" 06:24:03 DEBUG } 06:24:03 DEBUG ] 06:24:03 DEBUG ] 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:03 DEBUG ] 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:03 INFO [http_cmd]: http status code OK 06:24:03 DEBUG --- status code --- 06:24:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:03 DEBUG --- http response --- 06:24:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:24:03 DEBUG ----- output ----- 06:24:03 DEBUG { 06:24:03 DEBUG "status": "success", 06:24:03 DEBUG "data": { 06:24:03 DEBUG "resultType": "vector", 06:24:03 DEBUG "result": [ 06:24:03 DEBUG { 06:24:03 DEBUG "metric": { 06:24:03 DEBUG "instance": "10.142.0.11:9100" 06:24:03 DEBUG }, 06:24:03 DEBUG "value": [ 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG [...] 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG 06:24:03 DEBUG "instance": "10.142.0.51:9100" 06:24:03 DEBUG }, 06:24:03 DEBUG "value": [ 06:24:03 DEBUG 1735367043, 06:24:03 DEBUG "2402124.033333334" 06:24:03 DEBUG ] 06:24:03 DEBUG } 06:24:03 DEBUG ] 06:24:03 DEBUG } 06:24:03 DEBUG } 06:24:36 INFO 06:24:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:24:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:24:36 INFO [loop_until]: OK (rc = 0) 06:24:36 DEBUG --- stdout --- 06:24:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 709m 13412Mi rcs-687d95c5b-qs2xv 252m 427Mi rcs-687d95c5b-xxxmk 1m 396Mi 06:24:36 DEBUG --- stderr --- 06:24:36 DEBUG 06:24:38 INFO 06:24:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:24:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:24:38 INFO [loop_until]: OK (rc = 0) 06:24:38 DEBUG --- stdout --- 06:24:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 177m 1% 3463Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 150m 0% 18916Mi 66% 06:24:38 DEBUG --- stderr --- 06:24:38 DEBUG 06:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367102" 06:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367102" 06:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735367102" 06:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367102" 06:25:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367102" 06:25:02 INFO [http_cmd]: http status code OK 06:25:02 INFO [http_cmd]: http status code OK 06:25:02 INFO [http_cmd]: http status code OK 06:25:02 DEBUG --- status code --- 06:25:02 INFO [http_cmd]: http status code OK 06:25:02 INFO [http_cmd]: http status code OK 06:25:02 DEBUG --- status code --- 06:25:02 DEBUG --- status code --- 06:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:02 DEBUG --- status code --- 06:25:02 DEBUG --- status code --- 06:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:02 DEBUG --- http response --- 06:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:02 DEBUG --- http response --- 06:25:02 DEBUG --- http response --- 06:25:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:02 DEBUG --- http response --- 06:25:02 DEBUG --- http response --- 06:25:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 ERROR [http_cmd]: ERROR 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG --- status code --- 06:25:03 ERROR --- status code --- 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG --- http response --- 06:25:03 ERROR --- http response --- 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG --- http response --- 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG --- http response --- 06:25:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735367103" 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735367103" 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG { 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG "data": { 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG "result": [ 06:25:03 DEBUG { 06:25:03 DEBUG "metric": { 06:25:03 DEBUG "pod": "ds-idrepo-0" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG [...] 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG "832253952" 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:25:03 DEBUG { 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG "status": "success", 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367103" 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG { 06:25:03 DEBUG "data": { 06:25:03 DEBUG { 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG "data": { 06:25:03 DEBUG "result": [ 06:25:03 DEBUG "data": { 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG { 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG "result": [ 06:25:03 DEBUG "metric": { 06:25:03 DEBUG "result": [ 06:25:03 DEBUG { 06:25:03 DEBUG "instance": "10.142.0.11:9100" 06:25:03 DEBUG { 06:25:03 DEBUG "metric": { 06:25:03 DEBUG }, 06:25:03 DEBUG "metric": { 06:25:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:25:03 DEBUG "value": [ 06:25:03 DEBUG "instance": "10.142.0.11:9100" 06:25:03 DEBUG }, 06:25:03 DEBUG 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG [...] 06:25:03 DEBUG [...] 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG [...] 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG "instance": "10.142.0.51:9100" 06:25:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:25:03 DEBUG 06:25:03 DEBUG }, 06:25:03 DEBUG }, 06:25:03 DEBUG "instance": "10.142.0.51:9100" 06:25:03 DEBUG "value": [ 06:25:03 DEBUG "value": [ 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG "0.7402128945033988" 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG "19251.200000000004" 06:25:03 DEBUG "0.00033333333333303023" 06:25:03 DEBUG ] 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG ] 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 DEBUG ] 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG ] 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 DEBUG { 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG "data": { 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG "result": [ 06:25:03 DEBUG { 06:25:03 DEBUG "metric": { 06:25:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG [...] 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG "14915452928" 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG { 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG "data": { 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG "result": [ 06:25:03 DEBUG { 06:25:03 DEBUG "metric": { 06:25:03 DEBUG "instance": "10.142.0.11:9100" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG [...] 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG "instance": "10.142.0.51:9100" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG "2418903.6666666674" 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367103" 06:25:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367103" 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG { 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG "data": { 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG "result": [ 06:25:03 DEBUG { 06:25:03 DEBUG "metric": { 06:25:03 DEBUG "instance": "10.142.0.11:9100" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG [...] 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG "instance": "10.142.0.51:9100" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG "0.00040000000000191006" 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG { 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG "data": { 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG "result": [ 06:25:03 DEBUG { 06:25:03 DEBUG "metric": { 06:25:03 DEBUG "instance": "10.142.0.11:9100" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG [...] 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG "instance": "10.142.0.51:9100" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG "2401928.8000000007" 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG { 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG "data": { 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG "result": [ 06:25:03 DEBUG { 06:25:03 DEBUG "metric": { 06:25:03 DEBUG "pod": "ds-idrepo-0" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG [...] 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG "812.1323573755994" 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG { 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG "data": { 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG "result": [ 06:25:03 DEBUG { 06:25:03 INFO [http_cmd]: http status code OK 06:25:03 DEBUG "metric": { 06:25:03 DEBUG --- status code --- 06:25:03 DEBUG "pod": "ds-idrepo-0" 06:25:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:03 DEBUG }, 06:25:03 DEBUG --- http response --- 06:25:03 DEBUG "value": [ 06:25:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:25:03 DEBUG 06:25:03 DEBUG ----- output ----- 06:25:03 DEBUG 06:25:03 DEBUG { 06:25:03 DEBUG 06:25:03 DEBUG "status": "success", 06:25:03 DEBUG [...] 06:25:03 DEBUG "data": { 06:25:03 DEBUG 06:25:03 DEBUG "resultType": "vector", 06:25:03 DEBUG 06:25:03 DEBUG "result": [ 06:25:03 DEBUG 06:25:03 DEBUG { 06:25:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:25:03 DEBUG "metric": { 06:25:03 DEBUG }, 06:25:03 DEBUG "pod": "ds-idrepo-0" 06:25:03 DEBUG "value": [ 06:25:03 DEBUG }, 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG "0.0012427321970542035" 06:25:03 DEBUG 06:25:03 DEBUG ] 06:25:03 DEBUG 06:25:03 DEBUG } 06:25:03 DEBUG 06:25:03 DEBUG ] 06:25:03 DEBUG [...] 06:25:03 DEBUG } 06:25:03 DEBUG 06:25:03 DEBUG } 06:25:03 DEBUG 06:25:03 DEBUG 06:25:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:25:03 DEBUG }, 06:25:03 DEBUG "value": [ 06:25:03 DEBUG 1735367103, 06:25:03 DEBUG "0" 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG ] 06:25:03 DEBUG } 06:25:03 DEBUG } 06:25:36 INFO 06:25:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:25:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:25:36 INFO [loop_until]: OK (rc = 0) 06:25:36 DEBUG --- stdout --- 06:25:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 117m 13411Mi rcs-687d95c5b-qs2xv 153m 427Mi rcs-687d95c5b-xxxmk 1m 396Mi 06:25:36 DEBUG --- stderr --- 06:25:36 DEBUG 06:25:38 INFO 06:25:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:25:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:25:38 INFO [loop_until]: OK (rc = 0) 06:25:38 DEBUG --- stdout --- 06:25:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 204m 1% 3461Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 189m 0% 18910Mi 66% 06:25:38 DEBUG --- stderr --- 06:25:38 DEBUG 06:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367162" 06:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367162" 06:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735367162" 06:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367162" 06:26:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367162" 06:26:02 INFO [http_cmd]: http status code OK 06:26:02 DEBUG --- status code --- 06:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:02 DEBUG --- http response --- 06:26:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:02 INFO [http_cmd]: http status code OK 06:26:02 DEBUG --- status code --- 06:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:02 DEBUG --- http response --- 06:26:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:02 INFO [http_cmd]: http status code OK 06:26:02 DEBUG --- status code --- 06:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:02 DEBUG --- http response --- 06:26:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:02 INFO [http_cmd]: http status code OK 06:26:02 DEBUG --- status code --- 06:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:02 DEBUG --- http response --- 06:26:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:02 INFO [http_cmd]: http status code OK 06:26:02 DEBUG --- status code --- 06:26:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:02 DEBUG --- http response --- 06:26:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG --- status code --- 06:26:03 ERROR [http_cmd]: ERROR 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 ERROR --- status code --- 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG --- http response --- 06:26:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 ERROR --- http response --- 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735367163" 06:26:03 DEBUG --- status code --- 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367163" 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG { 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG "data": { 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG "result": [ 06:26:03 DEBUG { 06:26:03 DEBUG "metric": { 06:26:03 DEBUG "pod": "ds-idrepo-0" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG [...] 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:26:03 DEBUG }, 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG "value": [ 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG "832253952" 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG ] 06:26:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:03 DEBUG } 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367163" 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG } 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367163" 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG { 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG "data": { 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG "result": [ 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:26:03 DEBUG { 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG "metric": { 06:26:03 DEBUG { 06:26:03 DEBUG "instance": "10.142.0.11:9100" 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG }, 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG "data": { 06:26:03 DEBUG "value": [ 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG "result": [ 06:26:03 DEBUG 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:26:03 DEBUG { 06:26:03 DEBUG 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG "metric": { 06:26:03 DEBUG [...] 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG { 06:26:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:26:03 DEBUG 06:26:03 DEBUG { 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG }, 06:26:03 DEBUG 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG "data": { 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG "data": { 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG 06:26:03 DEBUG "instance": "10.142.0.51:9100" 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG "result": [ 06:26:03 DEBUG 06:26:03 DEBUG }, 06:26:03 DEBUG "result": [ 06:26:03 DEBUG { 06:26:03 DEBUG 06:26:03 DEBUG "value": [ 06:26:03 DEBUG { 06:26:03 DEBUG "metric": { 06:26:03 DEBUG [...] 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG "metric": { 06:26:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:26:03 DEBUG 06:26:03 DEBUG "0.0006666666666665342" 06:26:03 DEBUG "instance": "10.142.0.11:9100" 06:26:03 DEBUG }, 06:26:03 DEBUG 06:26:03 DEBUG ] 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG } 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:26:03 DEBUG ] 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG }, 06:26:03 DEBUG } 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG "value": [ 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG } 06:26:03 DEBUG 06:26:03 DEBUG [...] 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG [...] 06:26:03 DEBUG 06:26:03 DEBUG "0.09305393911222148" 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG ] 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG } 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:26:03 DEBUG 06:26:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:26:03 DEBUG ] 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG "instance": "10.142.0.51:9100" 06:26:03 DEBUG }, 06:26:03 DEBUG } 06:26:03 DEBUG { 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG } 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG "data": { 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG "14917816320" 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG "9693.866666666669" 06:26:03 DEBUG ] 06:26:03 DEBUG "result": [ 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG { 06:26:03 DEBUG } 06:26:03 DEBUG ] 06:26:03 DEBUG "metric": { 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG "instance": "10.142.0.11:9100" 06:26:03 DEBUG } 06:26:03 DEBUG } 06:26:03 DEBUG }, 06:26:03 DEBUG } 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG [...] 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG "instance": "10.142.0.51:9100" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG "1927894.7000000007" 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG } 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG { 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG "data": { 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG "result": [ 06:26:03 DEBUG { 06:26:03 DEBUG "metric": { 06:26:03 DEBUG "instance": "10.142.0.11:9100" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG [...] 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG "instance": "10.142.0.51:9100" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG "0.0002666666666688874" 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG } 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367163" 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG { 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG "data": { 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG "result": [ 06:26:03 DEBUG { 06:26:03 DEBUG "metric": { 06:26:03 DEBUG "pod": "ds-idrepo-0" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG [...] 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG "1153.2606612631243" 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG { 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG "data": { 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG "result": [ 06:26:03 DEBUG { 06:26:03 DEBUG "metric": { 06:26:03 DEBUG "instance": "10.142.0.11:9100" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG [...] 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG "instance": "10.142.0.51:9100" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG "1913172.0333333337" 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG { 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG "data": { 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG "result": [ 06:26:03 DEBUG { 06:26:03 DEBUG "metric": { 06:26:03 DEBUG "pod": "ds-idrepo-0" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG [...] 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG "0.0015399379985393713" 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG } 06:26:03 INFO [http_cmd]: http status code OK 06:26:03 DEBUG --- status code --- 06:26:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:03 DEBUG --- http response --- 06:26:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:26:03 DEBUG ----- output ----- 06:26:03 DEBUG { 06:26:03 DEBUG "status": "success", 06:26:03 DEBUG "data": { 06:26:03 DEBUG "resultType": "vector", 06:26:03 DEBUG "result": [ 06:26:03 DEBUG { 06:26:03 DEBUG "metric": { 06:26:03 DEBUG "pod": "ds-idrepo-0" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG [...] 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG 06:26:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:26:03 DEBUG }, 06:26:03 DEBUG "value": [ 06:26:03 DEBUG 1735367163, 06:26:03 DEBUG "0" 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG ] 06:26:03 DEBUG } 06:26:03 DEBUG } 06:26:36 INFO 06:26:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:26:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:26:36 INFO [loop_until]: OK (rc = 0) 06:26:36 DEBUG --- stdout --- 06:26:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 106m 13411Mi rcs-687d95c5b-qs2xv 126m 427Mi rcs-687d95c5b-xxxmk 1m 396Mi 06:26:36 DEBUG --- stderr --- 06:26:36 DEBUG 06:26:38 INFO 06:26:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:26:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:26:38 INFO [loop_until]: OK (rc = 0) 06:26:38 DEBUG --- stdout --- 06:26:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 229m 1% 3452Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 170m 0% 18912Mi 66% 06:26:38 DEBUG --- stderr --- 06:26:38 DEBUG 06:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367222" 06:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367222" 06:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367222" 06:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735367222" 06:27:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367222" 06:27:02 INFO [http_cmd]: http status code OK 06:27:02 INFO [http_cmd]: http status code OK 06:27:02 INFO [http_cmd]: http status code OK 06:27:02 DEBUG --- status code --- 06:27:02 INFO [http_cmd]: http status code OK 06:27:02 INFO [http_cmd]: http status code OK 06:27:02 DEBUG --- status code --- 06:27:02 DEBUG --- status code --- 06:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:02 DEBUG --- status code --- 06:27:02 DEBUG --- status code --- 06:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:02 DEBUG --- http response --- 06:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:02 DEBUG --- http response --- 06:27:02 DEBUG --- http response --- 06:27:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:02 DEBUG --- http response --- 06:27:02 DEBUG --- http response --- 06:27:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG --- http response --- 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG --- http response --- 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 ERROR [http_cmd]: ERROR 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- status code --- 06:27:03 ERROR --- status code --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG --- http response --- 06:27:03 ERROR --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:27:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG { 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG "data": { 06:27:03 DEBUG "resultType": "vector", 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367223" 06:27:03 DEBUG "result": [ 06:27:03 DEBUG { 06:27:03 DEBUG "metric": { 06:27:03 DEBUG "pod": "ds-idrepo-0" 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG [...] 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735367223" 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:27:03 DEBUG }, 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735367223" 06:27:03 DEBUG "value": [ 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367223" 06:27:03 DEBUG 1735367223, 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367223" 06:27:03 DEBUG "832253952" 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:27:03 DEBUG { 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG { 06:27:03 DEBUG { 06:27:03 DEBUG "data": { 06:27:03 DEBUG { 06:27:03 DEBUG { 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG "data": { 06:27:03 DEBUG "data": { 06:27:03 DEBUG "result": [ 06:27:03 DEBUG "data": { 06:27:03 DEBUG "data": { 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG { 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG "result": [ 06:27:03 DEBUG "result": [ 06:27:03 DEBUG "metric": { 06:27:03 DEBUG "result": [ 06:27:03 DEBUG "result": [ 06:27:03 DEBUG { 06:27:03 DEBUG { 06:27:03 DEBUG "instance": "10.142.0.11:9100" 06:27:03 DEBUG { 06:27:03 DEBUG { 06:27:03 DEBUG "metric": { 06:27:03 DEBUG "metric": { 06:27:03 DEBUG }, 06:27:03 DEBUG "metric": { 06:27:03 DEBUG "metric": { 06:27:03 DEBUG "instance": "10.142.0.11:9100" 06:27:03 DEBUG "instance": "10.142.0.11:9100" 06:27:03 DEBUG "value": [ 06:27:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:27:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:27:03 DEBUG }, 06:27:03 DEBUG }, 06:27:03 DEBUG 06:27:03 DEBUG }, 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 06:27:03 DEBUG "value": [ 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG [...] 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG [...] 06:27:03 DEBUG [...] 06:27:03 DEBUG 06:27:03 DEBUG [...] 06:27:03 DEBUG [...] 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG "instance": "10.142.0.51:9100" 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG }, 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367223" 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG "instance": "10.142.0.51:9100" 06:27:03 DEBUG "instance": "10.142.0.51:9100" 06:27:03 DEBUG "value": [ 06:27:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:27:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:27:03 DEBUG }, 06:27:03 DEBUG }, 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG }, 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 DEBUG "value": [ 06:27:03 DEBUG "1263558.1666666665" 06:27:03 DEBUG "value": [ 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG ] 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG "19933.866666666672" 06:27:03 DEBUG "0.0006666666666660605" 06:27:03 DEBUG } 06:27:03 DEBUG "14916198400" 06:27:03 DEBUG "0.03443567799199258" 06:27:03 DEBUG ] 06:27:03 DEBUG ] 06:27:03 DEBUG ] 06:27:03 DEBUG ] 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG ] 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG ] 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367223" 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG { 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG "data": { 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG "result": [ 06:27:03 DEBUG { 06:27:03 DEBUG "metric": { 06:27:03 DEBUG "instance": "10.142.0.11:9100" 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG [...] 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG "instance": "10.142.0.51:9100" 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG "0.0010000000000028803" 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367223" 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG { 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG "data": { 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG "result": [ 06:27:03 DEBUG { 06:27:03 DEBUG "metric": { 06:27:03 DEBUG "pod": "ds-idrepo-0" 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG [...] 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG "980.7171388538535" 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG { 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG "data": { 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG "result": [ 06:27:03 DEBUG { 06:27:03 DEBUG "metric": { 06:27:03 DEBUG "instance": "10.142.0.11:9100" 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG [...] 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:27:03 DEBUG 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG 06:27:03 DEBUG { 06:27:03 DEBUG 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG "instance": "10.142.0.51:9100" 06:27:03 DEBUG "data": { 06:27:03 DEBUG }, 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG "value": [ 06:27:03 DEBUG "result": [ 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG { 06:27:03 DEBUG "1246810.4000000001" 06:27:03 DEBUG "metric": { 06:27:03 DEBUG ] 06:27:03 DEBUG "pod": "ds-idrepo-0" 06:27:03 DEBUG } 06:27:03 DEBUG }, 06:27:03 DEBUG ] 06:27:03 DEBUG "value": [ 06:27:03 DEBUG } 06:27:03 DEBUG 06:27:03 DEBUG } 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG [...] 06:27:03 DEBUG 06:27:03 INFO [http_cmd]: http status code OK 06:27:03 DEBUG 06:27:03 DEBUG --- status code --- 06:27:03 DEBUG 06:27:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:27:03 DEBUG --- http response --- 06:27:03 DEBUG }, 06:27:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:27:03 DEBUG "value": [ 06:27:03 DEBUG ----- output ----- 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG { 06:27:03 DEBUG "0.0010869206920297411" 06:27:03 DEBUG "status": "success", 06:27:03 DEBUG ] 06:27:03 DEBUG "data": { 06:27:03 DEBUG } 06:27:03 DEBUG "resultType": "vector", 06:27:03 DEBUG ] 06:27:03 DEBUG "result": [ 06:27:03 DEBUG } 06:27:03 DEBUG { 06:27:03 DEBUG } 06:27:03 DEBUG "metric": { 06:27:03 DEBUG "pod": "ds-idrepo-0" 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG [...] 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG 06:27:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:27:03 DEBUG }, 06:27:03 DEBUG "value": [ 06:27:03 DEBUG 1735367223, 06:27:03 DEBUG "0" 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG ] 06:27:03 DEBUG } 06:27:03 DEBUG } 06:27:36 INFO 06:27:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:27:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:27:36 INFO [loop_until]: OK (rc = 0) 06:27:36 DEBUG --- stdout --- 06:27:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 76m 13412Mi rcs-687d95c5b-qs2xv 36m 427Mi rcs-687d95c5b-xxxmk 1m 396Mi 06:27:36 DEBUG --- stderr --- 06:27:36 DEBUG 06:27:38 INFO 06:27:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:27:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:27:38 INFO [loop_until]: OK (rc = 0) 06:27:38 DEBUG --- stdout --- 06:27:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 233m 1% 3456Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 164m 0% 18912Mi 66% 06:27:38 DEBUG --- stderr --- 06:27:38 DEBUG 06:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735367282" 06:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367282" 06:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367282" 06:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367282" 06:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367282" 06:28:02 INFO [http_cmd]: http status code OK 06:28:02 DEBUG --- status code --- 06:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:02 DEBUG --- http response --- 06:28:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:02 INFO [http_cmd]: http status code OK 06:28:02 INFO [http_cmd]: http status code OK 06:28:02 DEBUG --- status code --- 06:28:02 DEBUG --- status code --- 06:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:02 DEBUG --- http response --- 06:28:02 DEBUG --- http response --- 06:28:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:02 INFO [http_cmd]: http status code OK 06:28:02 INFO [http_cmd]: http status code OK 06:28:02 DEBUG --- status code --- 06:28:02 DEBUG --- status code --- 06:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:02 DEBUG --- http response --- 06:28:02 DEBUG --- http response --- 06:28:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735367283" 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG --- status code --- 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 ERROR [http_cmd]: ERROR 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG --- status code --- 06:28:03 ERROR --- status code --- 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG --- http response --- 06:28:03 ERROR --- http response --- 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:28:03 DEBUG --- status code --- 06:28:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG { 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG "data": { 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG "result": [ 06:28:03 DEBUG { 06:28:03 DEBUG "metric": { 06:28:03 DEBUG "pod": "ds-idrepo-0" 06:28:03 DEBUG }, 06:28:03 DEBUG "value": [ 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG [...] 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:28:03 DEBUG }, 06:28:03 DEBUG "value": [ 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG "832253952" 06:28:03 DEBUG ] 06:28:03 DEBUG } 06:28:03 DEBUG ] 06:28:03 DEBUG } 06:28:03 DEBUG } 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735367283" 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367283" 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG { 06:28:03 DEBUG { 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG "data": { 06:28:03 DEBUG "data": { 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG "result": [ 06:28:03 DEBUG "result": [ 06:28:03 DEBUG { 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG { 06:28:03 DEBUG "metric": { 06:28:03 DEBUG --- status code --- 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG "metric": { 06:28:03 DEBUG "instance": "10.142.0.11:9100" 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG "instance": "10.142.0.11:9100" 06:28:03 DEBUG }, 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG "value": [ 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:28:03 DEBUG }, 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:28:03 DEBUG 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG "value": [ 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:28:03 DEBUG { 06:28:03 DEBUG 06:28:03 DEBUG { 06:28:03 DEBUG 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG [...] 06:28:03 DEBUG { 06:28:03 DEBUG "data": { 06:28:03 DEBUG 06:28:03 DEBUG "data": { 06:28:03 DEBUG 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG [...] 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG 06:28:03 DEBUG "data": { 06:28:03 DEBUG "result": [ 06:28:03 DEBUG 06:28:03 DEBUG "result": [ 06:28:03 DEBUG 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG { 06:28:03 DEBUG 06:28:03 DEBUG { 06:28:03 DEBUG "instance": "10.142.0.51:9100" 06:28:03 DEBUG "result": [ 06:28:03 DEBUG "metric": { 06:28:03 DEBUG 06:28:03 DEBUG "metric": { 06:28:03 DEBUG }, 06:28:03 DEBUG { 06:28:03 DEBUG "instance": "10.142.0.11:9100" 06:28:03 DEBUG "instance": "10.142.0.51:9100" 06:28:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:28:03 DEBUG "value": [ 06:28:03 DEBUG "metric": { 06:28:03 DEBUG }, 06:28:03 DEBUG }, 06:28:03 DEBUG }, 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:28:03 DEBUG "value": [ 06:28:03 DEBUG "value": [ 06:28:03 DEBUG "value": [ 06:28:03 DEBUG "691247.7999999999" 06:28:03 DEBUG }, 06:28:03 DEBUG 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG 06:28:03 DEBUG ] 06:28:03 DEBUG "value": [ 06:28:03 DEBUG 06:28:03 DEBUG "0.0006666666666660605" 06:28:03 DEBUG 06:28:03 DEBUG } 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG ] 06:28:03 DEBUG 06:28:03 DEBUG ] 06:28:03 DEBUG 06:28:03 DEBUG [...] 06:28:03 DEBUG } 06:28:03 DEBUG [...] 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG } 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG ] 06:28:03 DEBUG } 06:28:03 DEBUG --- status code --- 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367283" 06:28:03 DEBUG } 06:28:03 DEBUG [...] 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG } 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367283" 06:28:03 DEBUG "instance": "10.142.0.51:9100" 06:28:03 DEBUG 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:28:03 DEBUG 06:28:03 DEBUG }, 06:28:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:28:03 DEBUG "value": [ 06:28:03 DEBUG }, 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG { 06:28:03 DEBUG }, 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG "value": [ 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG "value": [ 06:28:03 DEBUG "20070.4" 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:28:03 DEBUG "data": { 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG ] 06:28:03 DEBUG "0.10315591539925766" 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG "14919311360" 06:28:03 DEBUG } 06:28:03 DEBUG ] 06:28:03 DEBUG { 06:28:03 DEBUG "result": [ 06:28:03 DEBUG { 06:28:03 DEBUG ] 06:28:03 DEBUG } 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG ] 06:28:03 DEBUG "metric": { 06:28:03 DEBUG } 06:28:03 DEBUG ] 06:28:03 DEBUG "data": { 06:28:03 DEBUG } 06:28:03 DEBUG "instance": "10.142.0.11:9100" 06:28:03 DEBUG } 06:28:03 DEBUG } 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG ] 06:28:03 DEBUG }, 06:28:03 DEBUG } 06:28:03 DEBUG "result": [ 06:28:03 DEBUG } 06:28:03 DEBUG "value": [ 06:28:03 DEBUG { 06:28:03 DEBUG } 06:28:03 DEBUG 06:28:03 DEBUG "metric": { 06:28:03 DEBUG 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG "pod": "ds-idrepo-0" 06:28:03 DEBUG 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG }, 06:28:03 DEBUG [...] 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG "value": [ 06:28:03 DEBUG 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:28:03 DEBUG 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG "instance": "10.142.0.51:9100" 06:28:03 DEBUG { 06:28:03 DEBUG [...] 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG }, 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:28:03 DEBUG "value": [ 06:28:03 DEBUG "data": { 06:28:03 DEBUG 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG 06:28:03 DEBUG { 06:28:03 DEBUG "0.00036666666666936487" 06:28:03 DEBUG "result": [ 06:28:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG ] 06:28:03 DEBUG { 06:28:03 DEBUG }, 06:28:03 DEBUG "data": { 06:28:03 INFO [http_cmd]: http status code OK 06:28:03 DEBUG } 06:28:03 DEBUG "metric": { 06:28:03 DEBUG "value": [ 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG --- status code --- 06:28:03 DEBUG ] 06:28:03 DEBUG "instance": "10.142.0.11:9100" 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG "result": [ 06:28:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:03 DEBUG } 06:28:03 DEBUG }, 06:28:03 DEBUG "818.4388862802693" 06:28:03 DEBUG { 06:28:03 DEBUG --- http response --- 06:28:03 DEBUG } 06:28:03 DEBUG "value": [ 06:28:03 DEBUG ] 06:28:03 DEBUG "metric": { 06:28:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:28:03 DEBUG 06:28:03 DEBUG } 06:28:03 DEBUG "pod": "ds-idrepo-0" 06:28:03 DEBUG ----- output ----- 06:28:03 DEBUG 06:28:03 DEBUG ] 06:28:03 DEBUG }, 06:28:03 DEBUG { 06:28:03 DEBUG 06:28:03 DEBUG } 06:28:03 DEBUG "value": [ 06:28:03 DEBUG "status": "success", 06:28:03 DEBUG [...] 06:28:03 DEBUG } 06:28:03 DEBUG 06:28:03 DEBUG "data": { 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG "resultType": "vector", 06:28:03 DEBUG 06:28:03 DEBUG 06:28:03 DEBUG "result": [ 06:28:03 DEBUG 06:28:03 DEBUG [...] 06:28:03 DEBUG { 06:28:03 DEBUG "instance": "10.142.0.51:9100" 06:28:03 DEBUG 06:28:03 DEBUG "metric": { 06:28:03 DEBUG }, 06:28:03 DEBUG 06:28:03 DEBUG "pod": "ds-idrepo-0" 06:28:03 DEBUG "value": [ 06:28:03 DEBUG 06:28:03 DEBUG }, 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:28:03 DEBUG "value": [ 06:28:03 DEBUG "674241.1000000002" 06:28:03 DEBUG }, 06:28:03 DEBUG 06:28:03 DEBUG ] 06:28:03 DEBUG "value": [ 06:28:03 DEBUG 06:28:03 DEBUG } 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG 06:28:03 DEBUG ] 06:28:03 DEBUG "0.0013068820561552455" 06:28:03 DEBUG [...] 06:28:03 DEBUG } 06:28:03 DEBUG ] 06:28:03 DEBUG 06:28:03 DEBUG } 06:28:03 DEBUG } 06:28:03 DEBUG 06:28:03 DEBUG ] 06:28:03 DEBUG 06:28:03 DEBUG } 06:28:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:28:03 DEBUG } 06:28:03 DEBUG }, 06:28:03 DEBUG "value": [ 06:28:03 DEBUG 1735367283, 06:28:03 DEBUG "0" 06:28:03 DEBUG ] 06:28:03 DEBUG } 06:28:03 DEBUG ] 06:28:03 DEBUG } 06:28:03 DEBUG } 06:28:36 INFO 06:28:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:28:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:28:36 INFO [loop_until]: OK (rc = 0) 06:28:36 DEBUG --- stdout --- 06:28:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13411Mi rcs-687d95c5b-qs2xv 58m 427Mi rcs-687d95c5b-xxxmk 1m 396Mi 06:28:36 DEBUG --- stderr --- 06:28:36 DEBUG 06:28:38 INFO 06:28:38 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:28:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:28:39 INFO [loop_until]: OK (rc = 0) 06:28:39 DEBUG --- stdout --- 06:28:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 237m 1% 3453Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 771m 2% 18917Mi 66% 06:28:39 DEBUG --- stderr --- 06:28:39 DEBUG 06:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367342" 06:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735367342" 06:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367342" 06:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367342" 06:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367342" 06:29:02 INFO [http_cmd]: http status code OK 06:29:02 DEBUG --- status code --- 06:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:02 DEBUG --- http response --- 06:29:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:02 INFO [http_cmd]: http status code OK 06:29:02 DEBUG --- status code --- 06:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:02 DEBUG --- http response --- 06:29:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:02 INFO [http_cmd]: http status code OK 06:29:02 DEBUG --- status code --- 06:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:02 DEBUG --- http response --- 06:29:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:02 INFO [http_cmd]: http status code OK 06:29:02 DEBUG --- status code --- 06:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:02 DEBUG --- http response --- 06:29:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:02 INFO [http_cmd]: http status code OK 06:29:02 DEBUG --- status code --- 06:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:02 DEBUG --- http response --- 06:29:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735367343" 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 ERROR [http_cmd]: ERROR 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- status code --- 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 ERROR --- status code --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG --- status code --- 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 ERROR --- http response --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG { 06:29:03 DEBUG "status": "success", 06:29:03 DEBUG "data": { 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG "result": [ 06:29:03 DEBUG { 06:29:03 DEBUG "metric": { 06:29:03 DEBUG "pod": "ds-idrepo-0" 06:29:03 DEBUG }, 06:29:03 DEBUG "value": [ 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG [...] 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:29:03 DEBUG }, 06:29:03 DEBUG "value": [ 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG "832253952" 06:29:03 DEBUG ] 06:29:03 DEBUG } 06:29:03 DEBUG ] 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735367343" 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG { 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG "status": "success", 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG "data": { 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG "result": [ 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:29:03 DEBUG { 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG "metric": { 06:29:03 DEBUG { 06:29:03 DEBUG "instance": "10.142.0.11:9100" 06:29:03 DEBUG "status": "success", 06:29:03 DEBUG }, 06:29:03 DEBUG "data": { 06:29:03 DEBUG "value": [ 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG 06:29:03 DEBUG "result": [ 06:29:03 DEBUG 06:29:03 DEBUG { 06:29:03 DEBUG "metric": { 06:29:03 DEBUG 06:29:03 DEBUG "instance": "10.142.0.11:9100" 06:29:03 DEBUG [...] 06:29:03 DEBUG }, 06:29:03 DEBUG 06:29:03 DEBUG "value": [ 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "instance": "10.142.0.51:9100" 06:29:03 DEBUG 06:29:03 DEBUG }, 06:29:03 DEBUG [...] 06:29:03 DEBUG "value": [ 06:29:03 DEBUG 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG 06:29:03 DEBUG "0.00033333333333303023" 06:29:03 DEBUG 06:29:03 DEBUG ] 06:29:03 DEBUG "instance": "10.142.0.51:9100" 06:29:03 DEBUG } 06:29:03 DEBUG }, 06:29:03 DEBUG ] 06:29:03 DEBUG "value": [ 06:29:03 DEBUG } 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG } 06:29:03 DEBUG "94107.86666666668" 06:29:03 DEBUG ] 06:29:03 DEBUG } 06:29:03 DEBUG ] 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367343" 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 DEBUG --- status code --- 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367343" 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG { 06:29:03 DEBUG { 06:29:03 DEBUG { 06:29:03 DEBUG "status": "success", 06:29:03 DEBUG "status": "success", 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG "status": "success", 06:29:03 DEBUG "data": { 06:29:03 DEBUG "data": { 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG "data": { 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG "result": [ 06:29:03 DEBUG "result": [ 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG "result": [ 06:29:03 DEBUG { 06:29:03 DEBUG { 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:29:03 DEBUG { 06:29:03 DEBUG "metric": { 06:29:03 DEBUG "metric": { 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG "metric": { 06:29:03 DEBUG "instance": "10.142.0.11:9100" 06:29:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:29:03 DEBUG { 06:29:03 DEBUG { 06:29:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:29:03 DEBUG }, 06:29:03 DEBUG }, 06:29:03 DEBUG "status": "success", 06:29:03 DEBUG "status": "success", 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 INFO [http_cmd]: http status code OK 06:29:03 DEBUG }, 06:29:03 DEBUG "value": [ 06:29:03 DEBUG "value": [ 06:29:03 DEBUG "data": { 06:29:03 DEBUG "data": { 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG --- status code --- 06:29:03 DEBUG "value": [ 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "result": [ 06:29:03 DEBUG "result": [ 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG --- http response --- 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG { 06:29:03 DEBUG { 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:29:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:29:03 DEBUG 06:29:03 DEBUG [...] 06:29:03 DEBUG [...] 06:29:03 DEBUG "metric": { 06:29:03 DEBUG "metric": { 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG ----- output ----- 06:29:03 DEBUG [...] 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "instance": "10.142.0.11:9100" 06:29:03 DEBUG "pod": "ds-idrepo-0" 06:29:03 DEBUG { 06:29:03 DEBUG { 06:29:03 DEBUG { 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG }, 06:29:03 DEBUG }, 06:29:03 DEBUG "status": "success", 06:29:03 DEBUG "status": "success", 06:29:03 DEBUG "status": "success", 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "value": [ 06:29:03 DEBUG "value": [ 06:29:03 DEBUG "data": { 06:29:03 DEBUG "data": { 06:29:03 DEBUG "data": { 06:29:03 DEBUG 06:29:03 DEBUG "instance": "10.142.0.51:9100" 06:29:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG "resultType": "vector", 06:29:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:29:03 DEBUG }, 06:29:03 DEBUG }, 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "result": [ 06:29:03 DEBUG "result": [ 06:29:03 DEBUG "result": [ 06:29:03 DEBUG { 06:29:03 DEBUG "metric": { 06:29:03 DEBUG "instance": "10.142.0.11:9100" 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG { 06:29:03 DEBUG { 06:29:03 DEBUG }, 06:29:03 DEBUG "value": [ 06:29:03 DEBUG "value": [ 06:29:03 DEBUG }, 06:29:03 DEBUG [...] 06:29:03 DEBUG [...] 06:29:03 DEBUG "metric": { 06:29:03 DEBUG "metric": { 06:29:03 DEBUG "value": [ 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG "value": [ 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "pod": "ds-idrepo-0" 06:29:03 DEBUG "pod": "ds-idrepo-0" 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG "24985.600000000006" 06:29:03 DEBUG "14921203712" 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG }, 06:29:03 DEBUG }, 06:29:03 DEBUG "0.07119122576710228" 06:29:03 DEBUG ] 06:29:03 DEBUG ] 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "value": [ 06:29:03 DEBUG "value": [ 06:29:03 DEBUG ] 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG 06:29:03 DEBUG "instance": "10.142.0.51:9100" 06:29:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG } 06:29:03 DEBUG ] 06:29:03 DEBUG ] 06:29:03 DEBUG [...] 06:29:03 DEBUG }, 06:29:03 DEBUG }, 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG ] 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG 06:29:03 DEBUG "value": [ 06:29:03 DEBUG "value": [ 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG [...] 06:29:03 DEBUG [...] 06:29:03 DEBUG } 06:29:03 DEBUG 06:29:03 DEBUG "0.0005333333333396695" 06:29:03 DEBUG "1259.7053132629485" 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "instance": "10.142.0.51:9100" 06:29:03 DEBUG ] 06:29:03 DEBUG ] 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG }, 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG 06:29:03 DEBUG 06:29:03 DEBUG "value": [ 06:29:03 DEBUG ] 06:29:03 DEBUG ] 06:29:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:29:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG }, 06:29:03 DEBUG }, 06:29:03 DEBUG "73816.49999999999" 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG "value": [ 06:29:03 DEBUG "value": [ 06:29:03 DEBUG ] 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG 1735367343, 06:29:03 DEBUG } 06:29:03 DEBUG "0" 06:29:03 DEBUG "0.0012352106668075597" 06:29:03 DEBUG ] 06:29:03 DEBUG ] 06:29:03 DEBUG ] 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG ] 06:29:03 DEBUG ] 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:03 DEBUG } 06:29:36 INFO 06:29:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:29:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:29:36 INFO [loop_until]: OK (rc = 0) 06:29:36 DEBUG --- stdout --- 06:29:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 7m 13416Mi rcs-687d95c5b-qs2xv 46m 428Mi rcs-687d95c5b-xxxmk 1m 396Mi 06:29:36 DEBUG --- stderr --- 06:29:36 DEBUG 06:29:39 INFO 06:29:39 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:29:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:29:39 INFO [loop_until]: OK (rc = 0) 06:29:39 DEBUG --- stdout --- 06:29:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 225m 1% 3463Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 194m 0% 18910Mi 66% 06:29:39 DEBUG --- stderr --- 06:29:39 DEBUG 06:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735367402" 06:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367402" 06:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367402" 06:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367402" 06:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367402" 06:30:02 INFO [http_cmd]: http status code OK 06:30:02 INFO [http_cmd]: http status code OK 06:30:02 DEBUG --- status code --- 06:30:02 DEBUG --- status code --- 06:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:02 DEBUG --- http response --- 06:30:02 DEBUG --- http response --- 06:30:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:02 INFO [http_cmd]: http status code OK 06:30:02 DEBUG --- status code --- 06:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:02 DEBUG --- http response --- 06:30:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:02 INFO [http_cmd]: http status code OK 06:30:02 DEBUG --- status code --- 06:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:02 DEBUG --- http response --- 06:30:02 INFO [http_cmd]: http status code OK 06:30:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:02 DEBUG --- status code --- 06:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:02 DEBUG --- http response --- 06:30:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 ERROR [http_cmd]: ERROR 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 ERROR --- status code --- 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG --- status code --- 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 ERROR --- http response --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG { 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG "data": { 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG "result": [ 06:30:03 DEBUG { 06:30:03 DEBUG "metric": { 06:30:03 DEBUG "pod": "ds-idrepo-0" 06:30:03 DEBUG }, 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG [...] 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:30:03 DEBUG }, 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG "832253952" 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367403" 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:30:03 DEBUG { 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG { 06:30:03 DEBUG "data": { 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG "data": { 06:30:03 DEBUG "result": [ 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG { 06:30:03 DEBUG "result": [ 06:30:03 DEBUG "metric": { 06:30:03 DEBUG { 06:30:03 DEBUG "instance": "10.142.0.11:9100" 06:30:03 DEBUG "metric": { 06:30:03 DEBUG }, 06:30:03 DEBUG "instance": "10.142.0.11:9100" 06:30:03 DEBUG "value": [ 06:30:03 DEBUG }, 06:30:03 DEBUG 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG [...] 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG [...] 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "instance": "10.142.0.51:9100" 06:30:03 DEBUG 06:30:03 DEBUG }, 06:30:03 DEBUG "instance": "10.142.0.51:9100" 06:30:03 DEBUG "value": [ 06:30:03 DEBUG }, 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG "value": [ 06:30:03 DEBUG "92687.40000000001" 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG ] 06:30:03 DEBUG "0.001000000000001933" 06:30:03 DEBUG } 06:30:03 DEBUG ] 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367403" 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- http response --- 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG { 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:30:03 DEBUG "data": { 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG --- status code --- 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG "resultType": "vector", 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG { 06:30:03 DEBUG ----- output ----- 06:30:03 INFO [http_cmd]: http status code OK 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG "result": [ 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG { 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG { 06:30:03 DEBUG --- status code --- 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG { 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG "data": { 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG "data": { 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG "result": [ 06:30:03 DEBUG "metric": { 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG "data": { 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG { 06:30:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG "result": [ 06:30:03 DEBUG --- http response --- 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:30:03 DEBUG "metric": { 06:30:03 DEBUG }, 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG "result": [ 06:30:03 DEBUG { 06:30:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:30:03 DEBUG { 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG "pod": "ds-idrepo-0" 06:30:03 DEBUG "value": [ 06:30:03 DEBUG { 06:30:03 DEBUG { 06:30:03 DEBUG "metric": { 06:30:03 DEBUG ----- output ----- 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG { 06:30:03 DEBUG }, 06:30:03 DEBUG 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG "metric": { 06:30:03 DEBUG "instance": "10.142.0.11:9100" 06:30:03 DEBUG { 06:30:03 DEBUG "data": { 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 06:30:03 DEBUG "data": { 06:30:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:30:03 DEBUG }, 06:30:03 DEBUG "status": "success", 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG "data": { 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG }, 06:30:03 DEBUG "value": [ 06:30:03 DEBUG "data": { 06:30:03 DEBUG "result": [ 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG 06:30:03 DEBUG [...] 06:30:03 DEBUG "result": [ 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 06:30:03 DEBUG "resultType": "vector", 06:30:03 DEBUG { 06:30:03 DEBUG "result": [ 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG { 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "result": [ 06:30:03 DEBUG "metric": { 06:30:03 DEBUG { 06:30:03 DEBUG [...] 06:30:03 DEBUG 06:30:03 DEBUG "metric": { 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG { 06:30:03 DEBUG "pod": "ds-idrepo-0" 06:30:03 DEBUG "metric": { 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "instance": "10.142.0.11:9100" 06:30:03 DEBUG 06:30:03 DEBUG [...] 06:30:03 DEBUG "metric": { 06:30:03 DEBUG }, 06:30:03 DEBUG "instance": "10.142.0.11:9100" 06:30:03 DEBUG 06:30:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:30:03 DEBUG }, 06:30:03 DEBUG [...] 06:30:03 DEBUG 06:30:03 DEBUG "pod": "ds-idrepo-0" 06:30:03 DEBUG "value": [ 06:30:03 DEBUG }, 06:30:03 DEBUG 06:30:03 DEBUG }, 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG }, 06:30:03 DEBUG 06:30:03 DEBUG "value": [ 06:30:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG }, 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "instance": "10.142.0.51:9100" 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "value": [ 06:30:03 DEBUG "14924857344" 06:30:03 DEBUG 06:30:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:30:03 DEBUG }, 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG ] 06:30:03 DEBUG [...] 06:30:03 DEBUG }, 06:30:03 DEBUG [...] 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG "19933.86666666667" 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG ] 06:30:03 DEBUG 06:30:03 DEBUG "value": [ 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "805.3293456941245" 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 DEBUG 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG 06:30:03 DEBUG [...] 06:30:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:30:03 DEBUG ] 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG 06:30:03 DEBUG "0.095749824568602" 06:30:03 DEBUG [...] 06:30:03 DEBUG 06:30:03 DEBUG }, 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 DEBUG "instance": "10.142.0.51:9100" 06:30:03 DEBUG ] 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG "value": [ 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG }, 06:30:03 DEBUG } 06:30:03 DEBUG 06:30:03 DEBUG 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG } 06:30:03 DEBUG "value": [ 06:30:03 DEBUG ] 06:30:03 DEBUG 06:30:03 DEBUG "instance": "10.142.0.51:9100" 06:30:03 DEBUG "0.0013614007451210005" 06:30:03 DEBUG } 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG } 06:30:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:30:03 DEBUG }, 06:30:03 DEBUG ] 06:30:03 DEBUG "0.0005666666666646356" 06:30:03 DEBUG } 06:30:03 DEBUG }, 06:30:03 DEBUG "value": [ 06:30:03 DEBUG } 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG ] 06:30:03 DEBUG "value": [ 06:30:03 DEBUG ] 06:30:03 DEBUG "73230.16666666669" 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 DEBUG ] 06:30:03 DEBUG 1735367403, 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 DEBUG "0" 06:30:03 DEBUG ] 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:03 DEBUG ] 06:30:03 DEBUG } 06:30:03 DEBUG } 06:30:36 INFO 06:30:36 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 06:30:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:30:36 INFO [loop_until]: OK (rc = 0) 06:30:36 DEBUG --- stdout --- 06:30:36 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 532Mi ds-idrepo-0 48m 13417Mi rcs-687d95c5b-qs2xv 40m 428Mi rcs-687d95c5b-xxxmk 1m 396Mi 06:30:36 DEBUG --- stderr --- 06:30:36 DEBUG 06:30:39 INFO 06:30:39 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 06:30:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:30:39 INFO [loop_until]: OK (rc = 0) 06:30:39 DEBUG --- stdout --- 06:30:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 236m 1% 3473Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 198m 0% 18911Mi 66% 06:30:39 DEBUG --- stderr --- 06:30:39 DEBUG 06:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367462" 06:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735367462" 06:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367462" 06:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367462" 06:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367462" 06:31:02 INFO [http_cmd]: http status code OK 06:31:02 INFO [http_cmd]: http status code OK 06:31:02 INFO [http_cmd]: http status code OK 06:31:02 INFO [http_cmd]: http status code OK 06:31:02 INFO [http_cmd]: http status code OK 06:31:02 DEBUG --- status code --- 06:31:02 DEBUG --- status code --- 06:31:02 DEBUG --- status code --- 06:31:02 DEBUG --- status code --- 06:31:02 DEBUG --- status code --- 06:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:02 DEBUG --- http response --- 06:31:02 DEBUG --- http response --- 06:31:02 DEBUG --- http response --- 06:31:02 DEBUG --- http response --- 06:31:02 DEBUG --- http response --- 06:31:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 ERROR [http_cmd]: ERROR 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- status code --- 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 ERROR --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- http response --- 06:31:03 ERROR --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:31:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735367463" 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:31:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG { 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG "data": { 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG "result": [ 06:31:03 DEBUG { 06:31:03 DEBUG "metric": { 06:31:03 DEBUG "pod": "ds-idrepo-0" 06:31:03 DEBUG }, 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG [...] 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:31:03 DEBUG }, 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG "832253952" 06:31:03 DEBUG ] 06:31:03 DEBUG } 06:31:03 DEBUG ] 06:31:03 DEBUG } 06:31:03 DEBUG } 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367463" 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG { 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG "data": { 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG "result": [ 06:31:03 DEBUG { 06:31:03 DEBUG "metric": { 06:31:03 DEBUG "instance": "10.142.0.11:9100" 06:31:03 DEBUG }, 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG [...] 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG "instance": "10.142.0.51:9100" 06:31:03 DEBUG }, 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG "442869.6000000001" 06:31:03 DEBUG ] 06:31:03 DEBUG } 06:31:03 DEBUG ] 06:31:03 DEBUG } 06:31:03 DEBUG } 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG { 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG "data": { 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG "result": [ 06:31:03 DEBUG { 06:31:03 DEBUG "metric": { 06:31:03 DEBUG "instance": "10.142.0.11:9100" 06:31:03 DEBUG }, 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG [...] 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG "instance": "10.142.0.51:9100" 06:31:03 DEBUG }, 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG "0" 06:31:03 DEBUG ] 06:31:03 DEBUG } 06:31:03 DEBUG ] 06:31:03 DEBUG } 06:31:03 DEBUG } 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735367463" 06:31:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735367463" 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:31:03 DEBUG { 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG { 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:31:03 DEBUG "data": { 06:31:03 DEBUG { 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG "data": { 06:31:03 DEBUG { 06:31:03 DEBUG "result": [ 06:31:03 DEBUG "data": { 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG { 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG "result": [ 06:31:03 DEBUG "data": { 06:31:03 DEBUG "metric": { 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG "result": [ 06:31:03 DEBUG { 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG { 06:31:03 DEBUG "metric": { 06:31:03 DEBUG "result": [ 06:31:03 DEBUG }, 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG "metric": { 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG "instance": "10.142.0.11:9100" 06:31:03 DEBUG }, 06:31:03 DEBUG { 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:31:03 DEBUG "pod": "ds-idrepo-0" 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG "value": [ 06:31:03 DEBUG "value": [ 06:31:03 DEBUG "metric": { 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 INFO [http_cmd]: http status code OK 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG }, 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG --- status code --- 06:31:03 DEBUG { 06:31:03 DEBUG "value": [ 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG }, 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:31:03 DEBUG 06:31:03 DEBUG "value": [ 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG --- http response --- 06:31:03 DEBUG "data": { 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG [...] 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG [...] 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG [...] 06:31:03 DEBUG 06:31:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:31:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG "result": [ 06:31:03 DEBUG 06:31:03 DEBUG { 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG ----- output ----- 06:31:03 DEBUG }, 06:31:03 DEBUG { 06:31:03 DEBUG { 06:31:03 DEBUG 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG { 06:31:03 DEBUG "value": [ 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG "metric": { 06:31:03 DEBUG 06:31:03 DEBUG "data": { 06:31:03 DEBUG 06:31:03 DEBUG [...] 06:31:03 DEBUG "status": "success", 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG "data": { 06:31:03 DEBUG "pod": "ds-idrepo-0" 06:31:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG "instance": "10.142.0.51:9100" 06:31:03 DEBUG 06:31:03 DEBUG "data": { 06:31:03 DEBUG "505.3254924538388" 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG }, 06:31:03 DEBUG }, 06:31:03 DEBUG "result": [ 06:31:03 DEBUG }, 06:31:03 DEBUG 06:31:03 DEBUG "resultType": "vector", 06:31:03 DEBUG ] 06:31:03 DEBUG "result": [ 06:31:03 DEBUG "value": [ 06:31:03 DEBUG "value": [ 06:31:03 DEBUG { 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 06:31:03 DEBUG "result": [ 06:31:03 DEBUG } 06:31:03 DEBUG { 06:31:03 DEBUG 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG "metric": { 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 06:31:03 DEBUG { 06:31:03 DEBUG ] 06:31:03 DEBUG "metric": { 06:31:03 DEBUG 06:31:03 DEBUG "14920613888" 06:31:03 DEBUG "instance": "10.142.0.11:9100" 06:31:03 DEBUG "20206.933333333334" 06:31:03 DEBUG }, 06:31:03 DEBUG "metric": { 06:31:03 DEBUG } 06:31:03 DEBUG "pod": "ds-idrepo-0" 06:31:03 DEBUG 06:31:03 DEBUG ] 06:31:03 DEBUG }, 06:31:03 DEBUG ] 06:31:03 DEBUG "value": [ 06:31:03 DEBUG "instance": "10.142.0.11:9100" 06:31:03 DEBUG } 06:31:03 DEBUG }, 06:31:03 DEBUG [...] 06:31:03 DEBUG } 06:31:03 DEBUG "value": [ 06:31:03 DEBUG } 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG }, 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 06:31:03 DEBUG ] 06:31:03 DEBUG 06:31:03 DEBUG ] 06:31:03 DEBUG "0.07446626798898631" 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG } 06:31:03 DEBUG 06:31:03 DEBUG } 06:31:03 DEBUG ] 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG } 06:31:03 DEBUG 06:31:03 DEBUG } 06:31:03 DEBUG } 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:31:03 DEBUG [...] 06:31:03 DEBUG ] 06:31:03 DEBUG 06:31:03 DEBUG [...] 06:31:03 DEBUG }, 06:31:03 DEBUG 06:31:03 DEBUG } 06:31:03 DEBUG [...] 06:31:03 DEBUG 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG "0.0012732643069852752" 06:31:03 DEBUG "instance": "10.142.0.51:9100" 06:31:03 DEBUG 06:31:03 DEBUG 06:31:03 DEBUG } 06:31:03 DEBUG ] 06:31:03 DEBUG }, 06:31:03 DEBUG 06:31:03 DEBUG "pod": "rcs-687d95c5b-xxxmk" 06:31:03 DEBUG } 06:31:03 DEBUG "value": [ 06:31:03 DEBUG "instance": "10.142.0.51:9100" 06:31:03 DEBUG }, 06:31:03 DEBUG ] 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG }, 06:31:03 DEBUG "value": [ 06:31:03 DEBUG } 06:31:03 DEBUG "426357.00000000006" 06:31:03 DEBUG "value": [ 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG } 06:31:03 DEBUG ] 06:31:03 DEBUG 1735367463, 06:31:03 DEBUG "0" 06:31:03 DEBUG } 06:31:03 DEBUG "0.00053333333333588" 06:31:03 DEBUG ] 06:31:03 DEBUG ] 06:31:03 DEBUG ] 06:31:03 DEBUG } 06:31:03 DEBUG } 06:31:03 DEBUG } 06:31:03 DEBUG ] 06:31:03 DEBUG } 06:31:03 DEBUG ] 06:31:03 DEBUG } 06:31:03 DEBUG } 06:31:03 DEBUG } 06:31:03 DEBUG } 06:31:22 WARNING File /tmp/lodemon_data-20241227_183444/node_disk_read_bytes_total_file.json does not exist. Skipping average value computation... 127.0.0.1 - - [28/Dec/2024 06:31:22] "GET /monitoring/average?start_time=20241227_183747&stop_time=20241228_062921 HTTP/1.1" 200 - 06:31:33 INFO Finished: True 06:31:33 INFO Waiting for threads to register finish flag 06:32:03 INFO Done. Have a nice day! :) 127.0.0.1 - - [28/Dec/2024 06:32:03] "GET /monitoring/stop HTTP/1.1" 200 - 06:32:05 WARNING File /tmp/lodemon_data-20241227_183444/node_disk_read_bytes_total_file.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2646', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2891', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3450', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3352', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2744', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3087', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2205', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2695', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3136', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3401', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3038', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2842', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2793', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2156', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2450', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2499', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2352', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3254', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3303', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2303', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2401', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2254', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2940', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2548', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2597', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2989', ...) 127.0.0.1 - - [28/Dec/2024 06:32:06] "GET /monitoring/process HTTP/1.1" 200 -