==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-db8f6c7b5-rtlf5 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-frontend-18a44ad7-wxs9/10.142.0.11 Start Time: Sat, 28 Dec 2024 07:10:42 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=db8f6c7b5 skaffold.dev/run-id=c2900092-c265-43a0-8846-2ba6b9a48b22 Annotations: Status: Running IP: 10.106.40.190 IPs: IP: 10.106.40.190 Controlled By: ReplicaSet/lodemon-db8f6c7b5 Containers: lodemon: Container ID: containerd://ed6c76a9f3f1578052b43007a7af8a7fa8800bfce8366cba8755f5eb01cb6e56 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:4ba4f6066d128cbe60604a46e6c908ecfbc82c97c0cb359b8c9fdd92c9ab55a1 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sat, 28 Dec 2024 07:10:43 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: idc ds-only Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wvcvp (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-wvcvp: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 11:32:16 INFO 11:32:16 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:32:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:32:16 INFO [loop_until]: OK (rc = 0) 11:32:16 DEBUG --- stdout --- 11:32:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 105m 0% 3817Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 105m 0% 19057Mi 67% 11:32:16 DEBUG --- stderr --- 11:32:16 DEBUG 11:32:20 INFO 11:32:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:32:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:32:21 INFO [loop_until]: OK (rc = 0) 11:32:21 DEBUG --- stdout --- 11:32:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13478Mi rcs-645c86fdd6-j4rsk 54m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:32:21 DEBUG --- stderr --- 11:32:21 DEBUG 11:33:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735903992" 11:33:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903992" 11:33:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903992" 11:33:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903992" 11:33:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903992" 11:33:12 INFO [http_cmd]: http status code OK 11:33:12 INFO [http_cmd]: http status code OK 11:33:12 ERROR [http_cmd]: ERROR 11:33:12 INFO [http_cmd]: http status code OK 11:33:12 DEBUG --- status code --- 11:33:12 DEBUG --- status code --- 11:33:12 ERROR --- status code --- 11:33:12 DEBUG --- status code --- 11:33:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:12 DEBUG --- http response --- 11:33:12 INFO [http_cmd]: http status code OK 11:33:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:12 DEBUG --- http response --- 11:33:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:33:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:12 DEBUG --- status code --- 11:33:12 DEBUG --- http response --- 11:33:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:12 ERROR --- http response --- 11:33:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:33:12 DEBUG --- http response --- 11:33:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:33:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903993" 11:33:13 INFO [http_cmd]: http status code OK 11:33:13 DEBUG --- status code --- 11:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:13 DEBUG --- http response --- 11:33:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903993" 11:33:13 INFO [http_cmd]: http status code OK 11:33:13 DEBUG --- status code --- 11:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:13 DEBUG --- http response --- 11:33:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903994" 11:33:14 DEBUG --- status code --- 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG --- status code --- 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG --- http response --- 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735903994" 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG { 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG "data": { 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG "result": [ 11:33:14 DEBUG { 11:33:14 DEBUG "metric": { 11:33:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG [...] 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG "534.4816337182749" 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG { 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG "data": { 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG "result": [ 11:33:14 DEBUG { 11:33:14 DEBUG "metric": { 11:33:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG [...] 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG "0" 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG { 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG "data": { 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG "result": [ 11:33:14 DEBUG { 11:33:14 DEBUG "metric": { 11:33:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG [...] 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG "0.12769446075557472" 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735903994" 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG { 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG "data": { 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG "result": [ 11:33:14 DEBUG { 11:33:14 DEBUG "metric": { 11:33:14 DEBUG "instance": "10.142.0.11:9100" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG [...] 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG "instance": "10.142.0.51:9100" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG "1241553.2666666668" 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735903994" 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG { 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG "data": { 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG "result": [ 11:33:14 DEBUG { 11:33:14 DEBUG "metric": { 11:33:14 DEBUG "instance": "10.142.0.11:9100" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG [...] 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG "instance": "10.142.0.51:9100" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG "0.0003333333333330302" 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735903994" 11:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735903994" 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG { 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG "data": { 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG "result": [ 11:33:14 DEBUG { 11:33:14 DEBUG "metric": { 11:33:14 DEBUG "pod": "ds-idrepo-0" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG [...] 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG "758149120" 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 DEBUG --- status code --- 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG { 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 INFO [http_cmd]: http status code OK 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 DEBUG "data": { 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG --- status code --- 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG { 11:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:33:14 DEBUG { 11:33:14 DEBUG "result": [ 11:33:14 DEBUG { 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG --- http response --- 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG { 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG "data": { 11:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:33:14 DEBUG "data": { 11:33:14 DEBUG "metric": { 11:33:14 DEBUG "data": { 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG ----- output ----- 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG "instance": "10.142.0.11:9100" 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG "result": [ 11:33:14 DEBUG { 11:33:14 DEBUG "result": [ 11:33:14 DEBUG }, 11:33:14 DEBUG "result": [ 11:33:14 DEBUG { 11:33:14 DEBUG "status": "success", 11:33:14 DEBUG { 11:33:14 DEBUG "value": [ 11:33:14 DEBUG { 11:33:14 DEBUG "metric": { 11:33:14 DEBUG "data": { 11:33:14 DEBUG "metric": { 11:33:14 DEBUG 11:33:14 DEBUG "metric": { 11:33:14 DEBUG "instance": "10.142.0.11:9100" 11:33:14 DEBUG "resultType": "vector", 11:33:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:33:14 DEBUG 11:33:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:33:14 DEBUG }, 11:33:14 DEBUG "result": [ 11:33:14 DEBUG }, 11:33:14 DEBUG 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG { 11:33:14 DEBUG "value": [ 11:33:14 DEBUG [...] 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 11:33:14 DEBUG "metric": { 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG "instance": "10.142.0.11:9100" 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG }, 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG [...] 11:33:14 DEBUG "value": [ 11:33:14 DEBUG [...] 11:33:14 DEBUG "instance": "10.142.0.51:9100" 11:33:14 DEBUG [...] 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG }, 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG 11:33:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:33:14 DEBUG [...] 11:33:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:33:14 DEBUG "18295.466666666667" 11:33:14 DEBUG "instance": "10.142.0.51:9100" 11:33:14 DEBUG }, 11:33:14 DEBUG 11:33:14 DEBUG }, 11:33:14 DEBUG ] 11:33:14 DEBUG }, 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 11:33:14 DEBUG "value": [ 11:33:14 DEBUG } 11:33:14 DEBUG "value": [ 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG ] 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG "0.040251811862938865" 11:33:14 DEBUG "instance": "10.142.0.51:9100" 11:33:14 DEBUG "14990434304" 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG }, 11:33:14 DEBUG ] 11:33:14 DEBUG "1225077.2333333332" 11:33:14 DEBUG } 11:33:14 DEBUG "value": [ 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG ] 11:33:14 DEBUG 1735903994, 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 DEBUG "0.0004999999999995453" 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:14 DEBUG ] 11:33:14 DEBUG } 11:33:14 DEBUG } 11:33:16 INFO 11:33:16 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:33:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:33:16 INFO [loop_until]: OK (rc = 0) 11:33:16 DEBUG --- stdout --- 11:33:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3829Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 145m 0% 19055Mi 66% 11:33:16 DEBUG --- stderr --- 11:33:16 DEBUG 11:33:21 INFO 11:33:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:33:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:33:21 INFO [loop_until]: OK (rc = 0) 11:33:21 DEBUG --- stdout --- 11:33:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 30m 13477Mi rcs-645c86fdd6-j4rsk 80m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:33:21 DEBUG --- stderr --- 11:33:21 DEBUG 11:34:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904052" 11:34:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904052" 11:34:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904052" 11:34:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904052" 11:34:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904052" 11:34:12 INFO [http_cmd]: http status code OK 11:34:12 DEBUG --- status code --- 11:34:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:12 DEBUG --- http response --- 11:34:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:12 ERROR [http_cmd]: ERROR 11:34:12 INFO [http_cmd]: http status code OK 11:34:12 ERROR --- status code --- 11:34:12 DEBUG --- status code --- 11:34:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:34:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:12 ERROR --- http response --- 11:34:12 DEBUG --- http response --- 11:34:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:34:12 INFO [http_cmd]: http status code OK 11:34:12 INFO [http_cmd]: http status code OK 11:34:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:34:12 DEBUG --- status code --- 11:34:12 DEBUG --- status code --- 11:34:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:12 DEBUG --- http response --- 11:34:12 DEBUG --- http response --- 11:34:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904053" 11:34:13 INFO [http_cmd]: http status code OK 11:34:13 DEBUG --- status code --- 11:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:13 DEBUG --- http response --- 11:34:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904053" 11:34:13 INFO [http_cmd]: http status code OK 11:34:13 DEBUG --- status code --- 11:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:13 DEBUG --- http response --- 11:34:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG --- status code --- 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 DEBUG --- status code --- 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904054" 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- status code --- 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { 11:34:14 DEBUG { 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG "data": { 11:34:14 DEBUG "data": { 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG { 11:34:14 DEBUG "result": [ 11:34:14 DEBUG "result": [ 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG { 11:34:14 DEBUG { 11:34:14 DEBUG "data": { 11:34:14 DEBUG "metric": { 11:34:14 DEBUG "metric": { 11:34:14 DEBUG "pod": "ds-idrepo-0" 11:34:14 DEBUG "pod": "ds-idrepo-0" 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG }, 11:34:14 DEBUG }, 11:34:14 DEBUG "result": [ 11:34:14 DEBUG "value": [ 11:34:14 DEBUG "value": [ 11:34:14 DEBUG { 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "metric": { 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "pod": "ds-idrepo-0" 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG }, 11:34:14 DEBUG [...] 11:34:14 DEBUG [...] 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:34:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:34:14 DEBUG [...] 11:34:14 DEBUG }, 11:34:14 DEBUG }, 11:34:14 DEBUG 11:34:14 DEBUG "value": [ 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG 11:34:14 DEBUG "0.0005986101813775461" 11:34:14 DEBUG "0" 11:34:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:34:14 DEBUG ] 11:34:14 DEBUG ] 11:34:14 DEBUG }, 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 DEBUG "value": [ 11:34:14 DEBUG ] 11:34:14 DEBUG ] 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 DEBUG "391.6993401549202" 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG { 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG "data": { 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG "result": [ 11:34:14 DEBUG { 11:34:14 DEBUG "metric": { 11:34:14 DEBUG "instance": "10.142.0.11:9100" 11:34:14 DEBUG }, 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG [...] 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "instance": "10.142.0.51:9100" 11:34:14 DEBUG }, 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG "2163147.6999999997" 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG { 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG "data": { 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG "result": [ 11:34:14 DEBUG { 11:34:14 DEBUG "metric": { 11:34:14 DEBUG "instance": "10.142.0.11:9100" 11:34:14 DEBUG }, 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG [...] 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "instance": "10.142.0.51:9100" 11:34:14 DEBUG }, 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG "0.00033333333333492493" 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG { 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG "data": { 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG "result": [ 11:34:14 DEBUG { 11:34:14 DEBUG "metric": { 11:34:14 DEBUG "pod": "ds-idrepo-0" 11:34:14 DEBUG }, 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG [...] 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:34:14 DEBUG }, 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG "758157312" 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904054" 11:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904054" 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG { 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG "data": { 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG "result": [ 11:34:14 DEBUG { 11:34:14 DEBUG "metric": { 11:34:14 DEBUG "instance": "10.142.0.11:9100" 11:34:14 DEBUG }, 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG [...] 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "instance": "10.142.0.51:9100" 11:34:14 DEBUG }, 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG "9830.400000000001" 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- http response --- 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG { 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG "data": { 11:34:14 DEBUG ----- output ----- 11:34:14 INFO [http_cmd]: http status code OK 11:34:14 DEBUG { 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG { 11:34:14 DEBUG --- status code --- 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG "result": [ 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:34:14 DEBUG "data": { 11:34:14 DEBUG { 11:34:14 DEBUG "data": { 11:34:14 DEBUG --- http response --- 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG "metric": { 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:34:14 DEBUG "result": [ 11:34:14 DEBUG "instance": "10.142.0.11:9100" 11:34:14 DEBUG "result": [ 11:34:14 DEBUG ----- output ----- 11:34:14 DEBUG { 11:34:14 DEBUG }, 11:34:14 DEBUG { 11:34:14 DEBUG { 11:34:14 DEBUG "metric": { 11:34:14 DEBUG "value": [ 11:34:14 DEBUG "metric": { 11:34:14 DEBUG "status": "success", 11:34:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:34:14 DEBUG 11:34:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:34:14 DEBUG "data": { 11:34:14 DEBUG }, 11:34:14 DEBUG 11:34:14 DEBUG }, 11:34:14 DEBUG "resultType": "vector", 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG "value": [ 11:34:14 DEBUG "result": [ 11:34:14 DEBUG 11:34:14 DEBUG [...] 11:34:14 DEBUG 11:34:14 DEBUG { 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "metric": { 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "instance": "10.142.0.11:9100" 11:34:14 DEBUG [...] 11:34:14 DEBUG 11:34:14 DEBUG [...] 11:34:14 DEBUG }, 11:34:14 DEBUG 11:34:14 DEBUG "instance": "10.142.0.51:9100" 11:34:14 DEBUG 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG }, 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG 11:34:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:34:14 DEBUG 11:34:14 DEBUG }, 11:34:14 DEBUG "0.00043333333333066547" 11:34:14 DEBUG }, 11:34:14 DEBUG [...] 11:34:14 DEBUG "value": [ 11:34:14 DEBUG ] 11:34:14 DEBUG "value": [ 11:34:14 DEBUG 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG } 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG 11:34:14 DEBUG "14988124160" 11:34:14 DEBUG ] 11:34:14 DEBUG "0.042859950224812975" 11:34:14 DEBUG 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG ] 11:34:14 DEBUG "instance": "10.142.0.51:9100" 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 DEBUG }, 11:34:14 DEBUG ] 11:34:14 DEBUG ] 11:34:14 DEBUG "value": [ 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 DEBUG 1735904054, 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:14 DEBUG "2143967.9666666677" 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG ] 11:34:14 DEBUG } 11:34:14 DEBUG } 11:34:16 INFO 11:34:16 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:34:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:34:16 INFO [loop_until]: OK (rc = 0) 11:34:16 DEBUG --- stdout --- 11:34:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 226m 1% 3826Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 190m 0% 19052Mi 66% 11:34:16 DEBUG --- stderr --- 11:34:16 DEBUG 11:34:21 INFO 11:34:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:34:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:34:21 INFO [loop_until]: OK (rc = 0) 11:34:21 DEBUG --- stdout --- 11:34:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 160m 13477Mi rcs-645c86fdd6-j4rsk 111m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:34:21 DEBUG --- stderr --- 11:34:21 DEBUG 11:35:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904112" 11:35:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904112" 11:35:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904112" 11:35:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904112" 11:35:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904112" 11:35:13 INFO [http_cmd]: http status code OK 11:35:13 DEBUG --- status code --- 11:35:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:13 INFO [http_cmd]: http status code OK 11:35:13 INFO [http_cmd]: http status code OK 11:35:13 INFO [http_cmd]: http status code OK 11:35:13 DEBUG --- http response --- 11:35:13 ERROR [http_cmd]: ERROR 11:35:13 DEBUG --- status code --- 11:35:13 DEBUG --- status code --- 11:35:13 DEBUG --- status code --- 11:35:13 ERROR --- status code --- 11:35:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:35:13 DEBUG --- http response --- 11:35:13 DEBUG --- http response --- 11:35:13 DEBUG --- http response --- 11:35:13 ERROR --- http response --- 11:35:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:35:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:35:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904113" 11:35:13 INFO [http_cmd]: http status code OK 11:35:13 DEBUG --- status code --- 11:35:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:13 DEBUG --- http response --- 11:35:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904113" 11:35:13 INFO [http_cmd]: http status code OK 11:35:13 DEBUG --- status code --- 11:35:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:13 DEBUG --- http response --- 11:35:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904114" 11:35:14 DEBUG --- status code --- 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG --- status code --- 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG --- http response --- 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG --- http response --- 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904114" 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG { 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG { 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG "data": { 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:35:14 DEBUG "data": { 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG "result": [ 11:35:14 DEBUG { 11:35:14 DEBUG "result": [ 11:35:14 DEBUG { 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG { 11:35:14 DEBUG "metric": { 11:35:14 DEBUG "data": { 11:35:14 DEBUG "metric": { 11:35:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:35:14 DEBUG }, 11:35:14 DEBUG "result": [ 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG { 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904114" 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 11:35:14 DEBUG "metric": { 11:35:14 DEBUG 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG 11:35:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:35:14 DEBUG 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG 11:35:14 DEBUG }, 11:35:14 DEBUG 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG [...] 11:35:14 DEBUG "value": [ 11:35:14 DEBUG [...] 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG "pod": "ds-cts-0" 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG }, 11:35:14 DEBUG { 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG "value": [ 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG "pod": "ds-idrepo-0" 11:35:14 DEBUG [...] 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG "data": { 11:35:14 DEBUG }, 11:35:14 DEBUG 11:35:14 DEBUG "0" 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 11:35:14 DEBUG ] 11:35:14 DEBUG "result": [ 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG 11:35:14 DEBUG } 11:35:14 DEBUG { 11:35:14 DEBUG "2904760.5104362443" 11:35:14 DEBUG "pod": "ds-idrepo-0" 11:35:14 DEBUG ] 11:35:14 DEBUG "metric": { 11:35:14 DEBUG ] 11:35:14 DEBUG }, 11:35:14 DEBUG } 11:35:14 DEBUG "instance": "10.142.0.11:9100" 11:35:14 DEBUG } 11:35:14 DEBUG "value": [ 11:35:14 DEBUG } 11:35:14 DEBUG }, 11:35:14 DEBUG ] 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG } 11:35:14 DEBUG "1080.4893329376462" 11:35:14 DEBUG 11:35:14 DEBUG } 11:35:14 DEBUG ] 11:35:14 DEBUG 11:35:14 DEBUG } 11:35:14 DEBUG 11:35:14 DEBUG ] 11:35:14 DEBUG [...] 11:35:14 DEBUG } 11:35:14 DEBUG 11:35:14 DEBUG } 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG "instance": "10.142.0.51:9100" 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG "2365810.1666666665" 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG { 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG "data": { 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG "result": [ 11:35:14 DEBUG { 11:35:14 DEBUG "metric": { 11:35:14 DEBUG "instance": "10.142.0.11:9100" 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG [...] 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG "instance": "10.142.0.51:9100" 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG "0.0003333333333330302" 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904114" 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG { 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG "data": { 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG "result": [ 11:35:14 DEBUG { 11:35:14 DEBUG "metric": { 11:35:14 DEBUG "pod": "ds-idrepo-0" 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG [...] 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG "758149120" 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904114" 11:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904114" 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG { 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG "data": { 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG "result": [ 11:35:14 DEBUG { 11:35:14 DEBUG "metric": { 11:35:14 DEBUG "instance": "10.142.0.11:9100" 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG [...] 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG "instance": "10.142.0.51:9100" 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG "26350.93333333333" 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG { 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG "data": { 11:35:14 DEBUG { 11:35:14 DEBUG { 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG "result": [ 11:35:14 DEBUG "data": { 11:35:14 DEBUG "data": { 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG { 11:35:14 DEBUG "result": [ 11:35:14 DEBUG "result": [ 11:35:14 DEBUG "metric": { 11:35:14 DEBUG { 11:35:14 DEBUG { 11:35:14 DEBUG "instance": "10.142.0.11:9100" 11:35:14 DEBUG "metric": { 11:35:14 DEBUG "metric": { 11:35:14 DEBUG }, 11:35:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:35:14 DEBUG "instance": "10.142.0.11:9100" 11:35:14 DEBUG "value": [ 11:35:14 DEBUG }, 11:35:14 DEBUG }, 11:35:14 DEBUG 11:35:14 DEBUG "value": [ 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG [...] 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG [...] 11:35:14 DEBUG [...] 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG "instance": "10.142.0.51:9100" 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG }, 11:35:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:35:14 DEBUG "instance": "10.142.0.51:9100" 11:35:14 DEBUG "value": [ 11:35:14 DEBUG }, 11:35:14 DEBUG }, 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG "value": [ 11:35:14 DEBUG "0.0004333333333316129" 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG ] 11:35:14 DEBUG "14992244736" 11:35:14 DEBUG "2346488.0000000014" 11:35:14 DEBUG } 11:35:14 DEBUG ] 11:35:14 DEBUG ] 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 DEBUG ] 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:14 INFO [http_cmd]: http status code OK 11:35:14 DEBUG --- status code --- 11:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:35:14 DEBUG --- http response --- 11:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:35:14 DEBUG ----- output ----- 11:35:14 DEBUG { 11:35:14 DEBUG "status": "success", 11:35:14 DEBUG "data": { 11:35:14 DEBUG "resultType": "vector", 11:35:14 DEBUG "result": [ 11:35:14 DEBUG { 11:35:14 DEBUG "metric": { 11:35:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG [...] 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG 11:35:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:35:14 DEBUG }, 11:35:14 DEBUG "value": [ 11:35:14 DEBUG 1735904114, 11:35:14 DEBUG "0.03083124194472105" 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG ] 11:35:14 DEBUG } 11:35:14 DEBUG } 11:35:16 INFO 11:35:16 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:35:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:35:16 INFO [loop_until]: OK (rc = 0) 11:35:16 DEBUG --- stdout --- 11:35:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 220m 1% 3846Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 787m 2% 19055Mi 67% 11:35:16 DEBUG --- stderr --- 11:35:16 DEBUG 11:35:21 INFO 11:35:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:35:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:35:21 INFO [loop_until]: OK (rc = 0) 11:35:21 DEBUG --- stdout --- 11:35:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 679m 13478Mi rcs-645c86fdd6-j4rsk 129m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:35:21 DEBUG --- stderr --- 11:35:21 DEBUG 11:36:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904173" 11:36:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904173" 11:36:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904173" 11:36:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904173" 11:36:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904173" 11:36:13 INFO [http_cmd]: http status code OK 11:36:13 INFO [http_cmd]: http status code OK 11:36:13 ERROR [http_cmd]: ERROR 11:36:13 DEBUG --- status code --- 11:36:13 INFO [http_cmd]: http status code OK 11:36:13 DEBUG --- status code --- 11:36:13 INFO [http_cmd]: http status code OK 11:36:13 ERROR --- status code --- 11:36:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:13 DEBUG --- status code --- 11:36:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:13 DEBUG --- status code --- 11:36:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:36:13 DEBUG --- http response --- 11:36:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:13 DEBUG --- http response --- 11:36:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:13 ERROR --- http response --- 11:36:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:13 DEBUG --- http response --- 11:36:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:13 DEBUG --- http response --- 11:36:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:36:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:36:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904173" 11:36:13 INFO [http_cmd]: http status code OK 11:36:13 DEBUG --- status code --- 11:36:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:13 DEBUG --- http response --- 11:36:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904173" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 DEBUG --- status code --- 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904174" 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904174" 11:36:14 DEBUG --- http response --- 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904174" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904174" 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- status code --- 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904174" 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG --- status code --- 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG --- status code --- 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG { 11:36:14 DEBUG { 11:36:14 DEBUG { 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904174" 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG { 11:36:14 DEBUG { 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG "data": { 11:36:14 DEBUG "data": { 11:36:14 DEBUG "data": { 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG { 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG "data": { 11:36:14 DEBUG "data": { 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG "result": [ 11:36:14 DEBUG "result": [ 11:36:14 DEBUG "result": [ 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG "data": { 11:36:14 DEBUG { 11:36:14 DEBUG { 11:36:14 DEBUG { 11:36:14 DEBUG "result": [ 11:36:14 DEBUG "result": [ 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG "metric": { 11:36:14 DEBUG "metric": { 11:36:14 DEBUG "metric": { 11:36:14 DEBUG { 11:36:14 DEBUG { 11:36:14 DEBUG "result": [ 11:36:14 DEBUG "pod": "ds-cts-0" 11:36:14 DEBUG "pod": "ds-cts-0" 11:36:14 DEBUG "pod": "ds-cts-0" 11:36:14 DEBUG "metric": { 11:36:14 DEBUG "metric": { 11:36:14 DEBUG { 11:36:14 DEBUG }, 11:36:14 DEBUG }, 11:36:14 DEBUG }, 11:36:14 DEBUG "instance": "10.142.0.11:9100" 11:36:14 DEBUG "instance": "10.142.0.11:9100" 11:36:14 DEBUG "metric": { 11:36:14 DEBUG "value": [ 11:36:14 DEBUG "value": [ 11:36:14 DEBUG "value": [ 11:36:14 DEBUG }, 11:36:14 DEBUG }, 11:36:14 DEBUG "pod": "ds-idrepo-0" 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG "value": [ 11:36:14 DEBUG "value": [ 11:36:14 DEBUG }, 11:36:14 DEBUG "value": [ 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG [...] 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG [...] 11:36:14 DEBUG [...] 11:36:14 DEBUG [...] 11:36:14 DEBUG 11:36:14 DEBUG [...] 11:36:14 DEBUG [...] 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:36:14 DEBUG "pod": "ds-idrepo-0" 11:36:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:36:14 DEBUG }, 11:36:14 DEBUG "instance": "10.142.0.51:9100" 11:36:14 DEBUG "instance": "10.142.0.51:9100" 11:36:14 DEBUG }, 11:36:14 DEBUG }, 11:36:14 DEBUG }, 11:36:14 DEBUG "value": [ 11:36:14 DEBUG }, 11:36:14 DEBUG }, 11:36:14 DEBUG "value": [ 11:36:14 DEBUG "value": [ 11:36:14 DEBUG "value": [ 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG "value": [ 11:36:14 DEBUG "value": [ 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG "0" 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG "479.3433320069752" 11:36:14 DEBUG "0" 11:36:14 DEBUG "758149120" 11:36:14 DEBUG ] 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG "2369025.7666666666" 11:36:14 DEBUG "0.0003333333333330302" 11:36:14 DEBUG ] 11:36:14 DEBUG ] 11:36:14 DEBUG ] 11:36:14 DEBUG } 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG ] 11:36:14 DEBUG ] 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG ] 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG ] 11:36:14 DEBUG ] 11:36:14 DEBUG ] 11:36:14 DEBUG } 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG ] 11:36:14 DEBUG ] 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG { 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904174" 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904174" 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG "data": { 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904174" 11:36:14 DEBUG "resultType": "vector", 11:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904174" 11:36:14 DEBUG "result": [ 11:36:14 DEBUG { 11:36:14 DEBUG "metric": { 11:36:14 DEBUG "instance": "10.142.0.11:9100" 11:36:14 DEBUG }, 11:36:14 DEBUG "value": [ 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG [...] 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG "instance": "10.142.0.51:9100" 11:36:14 DEBUG }, 11:36:14 DEBUG "value": [ 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG "23620.266666666666" 11:36:14 DEBUG ] 11:36:14 DEBUG } 11:36:14 DEBUG ] 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG { 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG "data": { 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG "result": [ 11:36:14 DEBUG { 11:36:14 DEBUG "metric": { 11:36:14 DEBUG "instance": "10.142.0.11:9100" 11:36:14 DEBUG }, 11:36:14 DEBUG "value": [ 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG [...] 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG "instance": "10.142.0.51:9100" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG }, 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG "value": [ 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:36:14 DEBUG "2352465.033333334" 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG ] 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG { 11:36:14 DEBUG } 11:36:14 DEBUG { 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG ] 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG "data": { 11:36:14 DEBUG } 11:36:14 DEBUG "data": { 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG } 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG "result": [ 11:36:14 DEBUG "result": [ 11:36:14 DEBUG { 11:36:14 DEBUG { 11:36:14 DEBUG "metric": { 11:36:14 DEBUG "metric": { 11:36:14 DEBUG "instance": "10.142.0.11:9100" 11:36:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:36:14 INFO [http_cmd]: http status code OK 11:36:14 DEBUG }, 11:36:14 DEBUG }, 11:36:14 DEBUG --- status code --- 11:36:14 DEBUG "value": [ 11:36:14 DEBUG "value": [ 11:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG --- http response --- 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG ----- output ----- 11:36:14 DEBUG [...] 11:36:14 DEBUG [...] 11:36:14 DEBUG { 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG "status": "success", 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG "data": { 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG "resultType": "vector", 11:36:14 DEBUG "instance": "10.142.0.51:9100" 11:36:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:36:14 DEBUG "result": [ 11:36:14 DEBUG }, 11:36:14 DEBUG }, 11:36:14 DEBUG { 11:36:14 DEBUG "value": [ 11:36:14 DEBUG "value": [ 11:36:14 DEBUG "metric": { 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:36:14 DEBUG "0.00036666666666557526" 11:36:14 DEBUG "14989950976" 11:36:14 DEBUG }, 11:36:14 DEBUG ] 11:36:14 DEBUG ] 11:36:14 DEBUG "value": [ 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG 11:36:14 DEBUG ] 11:36:14 DEBUG ] 11:36:14 DEBUG 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:14 DEBUG [...] 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG 11:36:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:36:14 DEBUG }, 11:36:14 DEBUG "value": [ 11:36:14 DEBUG 1735904174, 11:36:14 DEBUG "0.413219335699807" 11:36:14 DEBUG ] 11:36:14 DEBUG } 11:36:14 DEBUG ] 11:36:14 DEBUG } 11:36:14 DEBUG } 11:36:16 INFO 11:36:16 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:36:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:36:17 INFO [loop_until]: OK (rc = 0) 11:36:17 DEBUG --- stdout --- 11:36:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 228m 1% 3849Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 204m 0% 19055Mi 67% 11:36:17 DEBUG --- stderr --- 11:36:17 DEBUG 11:36:21 INFO 11:36:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:36:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:36:21 INFO [loop_until]: OK (rc = 0) 11:36:21 DEBUG --- stdout --- 11:36:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 121m 13478Mi rcs-645c86fdd6-j4rsk 115m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:36:21 DEBUG --- stderr --- 11:36:21 DEBUG 11:37:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904233" 11:37:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904233" 11:37:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904233" 11:37:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904233" 11:37:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904233" 11:37:13 INFO [http_cmd]: http status code OK 11:37:13 DEBUG --- status code --- 11:37:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:13 DEBUG --- http response --- 11:37:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:13 INFO [http_cmd]: http status code OK 11:37:13 DEBUG --- status code --- 11:37:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:13 DEBUG --- http response --- 11:37:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:13 ERROR [http_cmd]: ERROR 11:37:13 ERROR --- status code --- 11:37:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:37:13 INFO [http_cmd]: http status code OK 11:37:13 ERROR --- http response --- 11:37:13 DEBUG --- status code --- 11:37:13 INFO [http_cmd]: http status code OK 11:37:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:37:13 DEBUG --- status code --- 11:37:13 DEBUG --- http response --- 11:37:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:37:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:13 DEBUG --- http response --- 11:37:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904233" 11:37:13 INFO [http_cmd]: http status code OK 11:37:13 DEBUG --- status code --- 11:37:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:13 DEBUG --- http response --- 11:37:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- status code --- 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- status code --- 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- http response --- 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG --- status code --- 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 DEBUG --- http response --- 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904234" 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG --- status code --- 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904234" 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904234" 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904234" 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- status code --- 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904234" 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:37:14 DEBUG ----- output ----- 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904234" 11:37:14 DEBUG ----- output ----- 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904234" 11:37:14 DEBUG --- status code --- 11:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904234" 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG ----- output ----- 11:37:14 DEBUG { 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG ----- output ----- 11:37:14 DEBUG { 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG { 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG { 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG "data": { 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG "data": { 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:37:14 DEBUG "data": { 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:37:14 DEBUG "data": { 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG ----- output ----- 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG ----- output ----- 11:37:14 DEBUG "result": [ 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG ----- output ----- 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG "result": [ 11:37:14 DEBUG { 11:37:14 DEBUG "result": [ 11:37:14 DEBUG { 11:37:14 DEBUG { 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG { 11:37:14 DEBUG "result": [ 11:37:14 DEBUG { 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG { 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG "metric": { 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG { 11:37:14 DEBUG "metric": { 11:37:14 DEBUG "data": { 11:37:14 DEBUG "metric": { 11:37:14 DEBUG "data": { 11:37:14 DEBUG "pod": "ds-idrepo-0" 11:37:14 DEBUG ----- output ----- 11:37:14 DEBUG "data": { 11:37:14 DEBUG "metric": { 11:37:14 DEBUG "pod": "ds-idrepo-0" 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG "pod": "ds-idrepo-0" 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG }, 11:37:14 DEBUG { 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG "pod": "ds-idrepo-0" 11:37:14 DEBUG }, 11:37:14 DEBUG "result": [ 11:37:14 DEBUG }, 11:37:14 DEBUG "result": [ 11:37:14 DEBUG "value": [ 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG "result": [ 11:37:14 DEBUG }, 11:37:14 DEBUG "value": [ 11:37:14 DEBUG { 11:37:14 DEBUG "value": [ 11:37:14 DEBUG { 11:37:14 DEBUG 11:37:14 DEBUG "data": { 11:37:14 DEBUG { 11:37:14 DEBUG "value": [ 11:37:14 DEBUG 11:37:14 DEBUG "metric": { 11:37:14 DEBUG 11:37:14 DEBUG "metric": { 11:37:14 DEBUG 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG "metric": { 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 INFO [http_cmd]: http status code OK 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG "instance": "10.142.0.11:9100" 11:37:14 DEBUG 11:37:14 DEBUG "instance": "10.142.0.11:9100" 11:37:14 DEBUG 11:37:14 DEBUG "result": [ 11:37:14 DEBUG "instance": "10.142.0.11:9100" 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG --- status code --- 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG }, 11:37:14 DEBUG 11:37:14 DEBUG }, 11:37:14 DEBUG [...] 11:37:14 DEBUG { 11:37:14 DEBUG }, 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:37:14 DEBUG 11:37:14 DEBUG [...] 11:37:14 DEBUG "value": [ 11:37:14 DEBUG [...] 11:37:14 DEBUG "value": [ 11:37:14 DEBUG 11:37:14 DEBUG "metric": { 11:37:14 DEBUG "value": [ 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG --- http response --- 11:37:14 DEBUG [...] 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG "instance": "10.142.0.11:9100" 11:37:14 DEBUG 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG }, 11:37:14 DEBUG 11:37:14 DEBUG ----- output ----- 11:37:14 DEBUG ----- output ----- 11:37:14 DEBUG ----- output ----- 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG "pod": "ds-cts-0" 11:37:14 DEBUG "value": [ 11:37:14 DEBUG 11:37:14 DEBUG { 11:37:14 DEBUG { 11:37:14 DEBUG { 11:37:14 DEBUG 11:37:14 DEBUG "pod": "ds-cts-0" 11:37:14 DEBUG [...] 11:37:14 DEBUG "pod": "ds-cts-0" 11:37:14 DEBUG [...] 11:37:14 DEBUG }, 11:37:14 DEBUG 11:37:14 DEBUG [...] 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG "status": "success", 11:37:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:37:14 DEBUG }, 11:37:14 DEBUG "value": [ 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG "0" 11:37:14 DEBUG ] 11:37:14 DEBUG } 11:37:14 DEBUG 11:37:14 DEBUG "data": { 11:37:14 DEBUG "data": { 11:37:14 DEBUG "data": { 11:37:14 DEBUG }, 11:37:14 DEBUG 11:37:14 DEBUG }, 11:37:14 DEBUG 11:37:14 DEBUG "value": [ 11:37:14 DEBUG ] 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG "resultType": "vector", 11:37:14 DEBUG "value": [ 11:37:14 DEBUG 11:37:14 DEBUG "value": [ 11:37:14 DEBUG 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG } 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG "result": [ 11:37:14 DEBUG "result": [ 11:37:14 DEBUG "result": [ 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG 11:37:14 DEBUG "0" 11:37:14 DEBUG ] 11:37:14 DEBUG [...] 11:37:14 DEBUG "instance": "10.142.0.51:9100" 11:37:14 DEBUG { 11:37:14 DEBUG { 11:37:14 DEBUG { 11:37:14 DEBUG "758149120" 11:37:14 DEBUG "instance": "10.142.0.51:9100" 11:37:14 DEBUG "0.0014342683357321492" 11:37:14 DEBUG ] 11:37:14 DEBUG "instance": "10.142.0.51:9100" 11:37:14 DEBUG } 11:37:14 DEBUG 11:37:14 DEBUG }, 11:37:14 DEBUG "metric": { 11:37:14 DEBUG "metric": { 11:37:14 DEBUG "metric": { 11:37:14 DEBUG ] 11:37:14 DEBUG }, 11:37:14 DEBUG } 11:37:14 DEBUG } 11:37:14 DEBUG }, 11:37:14 DEBUG ] 11:37:14 DEBUG 11:37:14 DEBUG "value": [ 11:37:14 DEBUG "instance": "10.142.0.11:9100" 11:37:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:37:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:37:14 DEBUG } 11:37:14 DEBUG "value": [ 11:37:14 DEBUG ] 11:37:14 DEBUG "value": [ 11:37:14 DEBUG } 11:37:14 DEBUG 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG }, 11:37:14 DEBUG }, 11:37:14 DEBUG }, 11:37:14 DEBUG ] 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG } 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG } 11:37:14 DEBUG "instance": "10.142.0.51:9100" 11:37:14 DEBUG "0.001000000000000038" 11:37:14 DEBUG "value": [ 11:37:14 DEBUG "value": [ 11:37:14 DEBUG "value": [ 11:37:14 DEBUG } 11:37:14 DEBUG "1227707.0000000005" 11:37:14 DEBUG } 11:37:14 DEBUG "30583.466666666667" 11:37:14 DEBUG }, 11:37:14 DEBUG ] 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG } 11:37:14 DEBUG ] 11:37:14 DEBUG ] 11:37:14 DEBUG "value": [ 11:37:14 DEBUG } 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG } 11:37:14 DEBUG } 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG ] 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG ] 11:37:14 DEBUG ] 11:37:14 DEBUG "1208761.033333333" 11:37:14 DEBUG } 11:37:14 DEBUG [...] 11:37:14 DEBUG [...] 11:37:14 DEBUG [...] 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG } 11:37:14 DEBUG } 11:37:14 DEBUG ] 11:37:14 DEBUG } 11:37:14 DEBUG }, 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG } 11:37:14 DEBUG } 11:37:14 DEBUG } 11:37:14 DEBUG "value": [ 11:37:14 DEBUG 11:37:14 DEBUG 11:37:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:37:14 DEBUG }, 11:37:14 DEBUG "instance": "10.142.0.51:9100" 11:37:14 DEBUG ] 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG "value": [ 11:37:14 DEBUG }, 11:37:14 DEBUG } 11:37:14 DEBUG "14990102528" 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG "value": [ 11:37:14 DEBUG } 11:37:14 DEBUG ] 11:37:14 DEBUG "0.12790837589315668" 11:37:14 DEBUG 1735904234, 11:37:14 DEBUG } 11:37:14 DEBUG ] 11:37:14 DEBUG "0.001266666666663241" 11:37:14 DEBUG ] 11:37:14 DEBUG } 11:37:14 DEBUG ] 11:37:14 DEBUG } 11:37:14 DEBUG ] 11:37:14 DEBUG } 11:37:14 DEBUG } 11:37:14 DEBUG } 11:37:14 DEBUG ] 11:37:14 DEBUG } 11:37:14 DEBUG } 11:37:14 DEBUG } 11:37:17 INFO 11:37:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:37:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:37:17 INFO [loop_until]: OK (rc = 0) 11:37:17 DEBUG --- stdout --- 11:37:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 170m 1% 3852Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 128m 0% 19055Mi 67% 11:37:17 DEBUG --- stderr --- 11:37:17 DEBUG 11:37:21 INFO 11:37:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:37:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:37:21 INFO [loop_until]: OK (rc = 0) 11:37:21 DEBUG --- stdout --- 11:37:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 66m 13478Mi rcs-645c86fdd6-j4rsk 108m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:37:21 DEBUG --- stderr --- 11:37:21 DEBUG 11:38:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904293" 11:38:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904293" 11:38:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904293" 11:38:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904293" 11:38:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904293" 11:38:13 ERROR [http_cmd]: ERROR 11:38:13 ERROR --- status code --- 11:38:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:38:13 ERROR --- http response --- 11:38:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:38:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:38:13 INFO [http_cmd]: http status code OK 11:38:13 DEBUG --- status code --- 11:38:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:13 INFO [http_cmd]: http status code OK 11:38:13 DEBUG --- http response --- 11:38:13 DEBUG --- status code --- 11:38:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:13 DEBUG --- http response --- 11:38:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:13 INFO [http_cmd]: http status code OK 11:38:13 DEBUG --- status code --- 11:38:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:13 DEBUG --- http response --- 11:38:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:13 INFO [http_cmd]: http status code OK 11:38:13 DEBUG --- status code --- 11:38:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:13 DEBUG --- http response --- 11:38:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904293" 11:38:13 INFO [http_cmd]: http status code OK 11:38:13 DEBUG --- status code --- 11:38:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:13 DEBUG --- http response --- 11:38:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904294" 11:38:14 DEBUG --- status code --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG --- status code --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG --- http response --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- http response --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG --- status code --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904294" 11:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904294" 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735904294, "407.819189539013" ] } ] } } 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG --- http response --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG ----- output ----- 11:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735904294, "0" ] } ] } } 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:38:14 DEBUG ----- output ----- 11:38:14 DEBUG { 11:38:14 DEBUG "status": "success", 11:38:14 DEBUG "data": { 11:38:14 DEBUG "resultType": "vector", 11:38:14 DEBUG "result": [ 11:38:14 DEBUG { 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG { 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG "status": "success", 11:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:38:14 DEBUG "metric": { 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG --- status code --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG "data": { 11:38:14 DEBUG ----- output ----- 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG ----- output ----- 11:38:14 DEBUG "pod": "ds-cts-0" 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG "resultType": "vector", 11:38:14 DEBUG { 11:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:38:14 DEBUG { 11:38:14 DEBUG }, 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG --- http response --- 11:38:14 INFO [http_cmd]: http status code OK 11:38:14 DEBUG "status": "success", 11:38:14 DEBUG ----- output ----- 11:38:14 DEBUG "status": "success", 11:38:14 DEBUG "value": [ 11:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:38:14 DEBUG "result": [ 11:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:38:14 DEBUG ----- output ----- 11:38:14 DEBUG { 11:38:14 DEBUG "status": "success", 11:38:14 DEBUG "data": { 11:38:14 DEBUG "resultType": "vector", 11:38:14 DEBUG "result": [ 11:38:14 DEBUG { 11:38:14 DEBUG "metric": { 11:38:14 DEBUG "instance": "10.142.0.11:9100" 11:38:14 DEBUG --- status code --- 11:38:14 DEBUG "data": { 11:38:14 DEBUG { 11:38:14 DEBUG "data": { 11:38:14 DEBUG 11:38:14 DEBUG ----- output ----- 11:38:14 DEBUG { 11:38:14 DEBUG }, 11:38:14 DEBUG "value": [ 11:38:14 DEBUG 11:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:38:14 DEBUG "status": "success", 11:38:14 DEBUG "resultType": "vector", 11:38:14 DEBUG 11:38:14 DEBUG { 11:38:14 DEBUG "metric": { 11:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:38:14 DEBUG "resultType": "vector", 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG [...] 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG "status": "success", 11:38:14 DEBUG "pod": "ds-idrepo-0" 11:38:14 DEBUG ----- output ----- 11:38:14 DEBUG "result": [ 11:38:14 DEBUG --- http response --- 11:38:14 DEBUG "data": { 11:38:14 DEBUG "result": [ 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG "data": { 11:38:14 DEBUG }, 11:38:14 DEBUG { 11:38:14 DEBUG { 11:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:38:14 DEBUG "resultType": "vector", 11:38:14 DEBUG { 11:38:14 DEBUG [...] 11:38:14 DEBUG "instance": "10.142.0.51:9100" 11:38:14 DEBUG "resultType": "vector", 11:38:14 DEBUG "value": [ 11:38:14 DEBUG "status": "success", 11:38:14 DEBUG "metric": { 11:38:14 DEBUG ----- output ----- 11:38:14 DEBUG "result": [ 11:38:14 DEBUG "metric": { 11:38:14 DEBUG 11:38:14 DEBUG }, 11:38:14 DEBUG "value": [ 11:38:14 DEBUG 1735904294, 11:38:14 DEBUG "0.0003000000000004851" 11:38:14 DEBUG ] 11:38:14 DEBUG } 11:38:14 DEBUG ] 11:38:14 DEBUG { 11:38:14 DEBUG 11:38:14 DEBUG "result": [ 11:38:14 DEBUG 11:38:14 DEBUG "data": { 11:38:14 DEBUG "instance": "10.142.0.11:9100" 11:38:14 DEBUG { 11:38:14 DEBUG "instance": "10.142.0.11:9100" 11:38:14 DEBUG } 11:38:14 DEBUG } 11:38:14 DEBUG 11:38:14 DEBUG { 11:38:14 DEBUG 11:38:14 DEBUG "resultType": "vector", 11:38:14 DEBUG }, 11:38:14 DEBUG "status": "success", 11:38:14 DEBUG }, 11:38:14 DEBUG "metric": { 11:38:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:38:14 DEBUG "metric": { 11:38:14 DEBUG 11:38:14 DEBUG "result": [ 11:38:14 DEBUG "value": [ 11:38:14 DEBUG "data": { 11:38:14 DEBUG "value": [ 11:38:14 DEBUG 11:38:14 DEBUG "instance": "10.142.0.11:9100" 11:38:14 DEBUG "instance": "10.142.0.11:9100" 11:38:14 DEBUG [...] 11:38:14 DEBUG { 11:38:14 DEBUG 11:38:14 DEBUG "resultType": "vector", 11:38:14 DEBUG }, 11:38:14 DEBUG 11:38:14 DEBUG }, 11:38:14 DEBUG }, 11:38:14 DEBUG 11:38:14 DEBUG "metric": { 11:38:14 DEBUG 11:38:14 DEBUG "result": [ 11:38:14 DEBUG "value": [ 11:38:14 DEBUG 11:38:14 DEBUG "value": [ 11:38:14 DEBUG "value": [ 11:38:14 DEBUG 11:38:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:38:14 DEBUG 11:38:14 DEBUG { 11:38:14 DEBUG 1735904294, 11:38:14 DEBUG [...] 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG }, 11:38:14 DEBUG [...] 11:38:14 DEBUG "metric": { 11:38:14 DEBUG "0" 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:38:14 DEBUG "value": [ 11:38:14 DEBUG 11:38:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:38:14 DEBUG ] 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG }, 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG }, 11:38:14 DEBUG } 11:38:14 DEBUG 11:38:14 DEBUG [...] 11:38:14 DEBUG [...] 11:38:14 DEBUG "value": [ 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG "value": [ 11:38:14 DEBUG ] 11:38:14 DEBUG "instance": "10.142.0.51:9100" 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG 1735904294, 11:38:14 DEBUG 11:38:14 DEBUG "instance": "10.142.0.51:9100" 11:38:14 DEBUG 11:38:14 DEBUG } 11:38:14 DEBUG }, 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG "758149120" 11:38:14 DEBUG [...] 11:38:14 DEBUG }, 11:38:14 DEBUG 11:38:14 DEBUG } 11:38:14 DEBUG "value": [ 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG ] 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG 11:38:14 DEBUG 1735904294, 11:38:14 DEBUG "instance": "10.142.0.51:9100" 11:38:14 DEBUG "instance": "10.142.0.51:9100" 11:38:14 DEBUG } 11:38:14 DEBUG "value": [ 11:38:14 DEBUG 11:38:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:38:14 DEBUG "0.0003333333333330302" 11:38:14 DEBUG }, 11:38:14 DEBUG }, 11:38:14 DEBUG ] 11:38:14 DEBUG 1735904294, 11:38:14 DEBUG [...] 11:38:14 DEBUG }, 11:38:14 DEBUG ] 11:38:14 DEBUG "value": [ 11:38:14 DEBUG "value": [ 11:38:14 DEBUG } 11:38:14 DEBUG "1230438.1333333333" 11:38:14 DEBUG 11:38:14 DEBUG "value": [ 11:38:14 DEBUG } 11:38:14 DEBUG ] 11:38:14 DEBUG 1735904294, 11:38:14 DEBUG } 11:38:14 DEBUG ] 11:38:14 DEBUG } 11:38:14 DEBUG ] 11:38:14 DEBUG 1735904294, 11:38:14 DEBUG } 11:38:14 DEBUG "1213142.066666667" 11:38:14 DEBUG 11:38:14 DEBUG } 11:38:14 DEBUG 1735904294, 11:38:14 DEBUG "14990131200" 11:38:14 DEBUG } 11:38:14 DEBUG ] 11:38:14 DEBUG } 11:38:14 DEBUG ] 11:38:14 DEBUG } 11:38:14 DEBUG } 11:38:14 DEBUG 11:38:14 DEBUG } 11:38:14 DEBUG "19524.266666666666" 11:38:14 DEBUG ] 11:38:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:38:14 DEBUG ] 11:38:14 DEBUG } 11:38:14 DEBUG }, 11:38:14 DEBUG } 11:38:14 DEBUG ] 11:38:14 DEBUG } 11:38:14 DEBUG } 11:38:14 DEBUG "value": [ 11:38:14 DEBUG ] 11:38:14 DEBUG 1735904294, 11:38:14 DEBUG } 11:38:14 DEBUG "0.1688944599300869" 11:38:14 DEBUG } 11:38:14 DEBUG ] 11:38:14 DEBUG } 11:38:14 DEBUG ] 11:38:14 DEBUG } 11:38:14 DEBUG } 11:38:17 INFO 11:38:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:38:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:38:17 INFO [loop_until]: OK (rc = 0) 11:38:17 DEBUG --- stdout --- 11:38:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 169m 1% 3854Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 132m 0% 19056Mi 67% 11:38:17 DEBUG --- stderr --- 11:38:17 DEBUG 11:38:21 INFO 11:38:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:38:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:38:21 INFO [loop_until]: OK (rc = 0) 11:38:21 DEBUG --- stdout --- 11:38:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 61m 13478Mi rcs-645c86fdd6-j4rsk 49m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:38:21 DEBUG --- stderr --- 11:38:21 DEBUG 11:39:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904353" 11:39:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904353" 11:39:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904353" 11:39:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904353" 11:39:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904353" 11:39:13 INFO [http_cmd]: http status code OK 11:39:13 DEBUG --- status code --- 11:39:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:13 DEBUG --- http response --- 11:39:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:13 INFO [http_cmd]: http status code OK 11:39:13 DEBUG --- status code --- 11:39:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:13 DEBUG --- http response --- 11:39:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:13 INFO [http_cmd]: http status code OK 11:39:13 DEBUG --- status code --- 11:39:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:13 DEBUG --- http response --- 11:39:13 INFO [http_cmd]: http status code OK 11:39:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:13 ERROR [http_cmd]: ERROR 11:39:13 DEBUG --- status code --- 11:39:13 ERROR --- status code --- 11:39:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:39:13 DEBUG --- http response --- 11:39:13 ERROR --- http response --- 11:39:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:39:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:39:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904353" 11:39:13 INFO [http_cmd]: http status code OK 11:39:13 DEBUG --- status code --- 11:39:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:13 DEBUG --- http response --- 11:39:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904354" 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG { 11:39:14 DEBUG { 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG "data": { 11:39:14 DEBUG "data": { 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG "result": [ 11:39:14 DEBUG "result": [ 11:39:14 DEBUG { 11:39:14 DEBUG { 11:39:14 DEBUG "metric": { 11:39:14 DEBUG "metric": { 11:39:14 DEBUG "pod": "ds-idrepo-0" 11:39:14 DEBUG "pod": "ds-idrepo-0" 11:39:14 DEBUG }, 11:39:14 DEBUG }, 11:39:14 DEBUG "value": [ 11:39:14 DEBUG "value": [ 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG [...] 11:39:14 DEBUG [...] 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG "pod": "ds-cts-0" 11:39:14 DEBUG "pod": "ds-cts-0" 11:39:14 DEBUG }, 11:39:14 DEBUG }, 11:39:14 DEBUG "value": [ 11:39:14 DEBUG "value": [ 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG "0" 11:39:14 DEBUG "420.3278324943459" 11:39:14 DEBUG ] 11:39:14 DEBUG ] 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG ] 11:39:14 DEBUG ] 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904354" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904354" 11:39:14 DEBUG --- status code --- 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904354" 11:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904354" 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG { 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG "data": { 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG "result": [ 11:39:14 DEBUG { 11:39:14 DEBUG "metric": { 11:39:14 DEBUG "instance": "10.142.0.11:9100" 11:39:14 DEBUG }, 11:39:14 DEBUG "value": [ 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG [...] 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG "instance": "10.142.0.51:9100" 11:39:14 DEBUG }, 11:39:14 DEBUG "value": [ 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG "0.00046666666666700016" 11:39:14 DEBUG ] 11:39:14 DEBUG } 11:39:14 DEBUG ] 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:39:14 DEBUG { 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG { 11:39:14 DEBUG "data": { 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG "data": { 11:39:14 DEBUG "result": [ 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG "resultType": "vector", 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG { 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG "result": [ 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG "metric": { 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG { 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG "pod": "ds-idrepo-0" 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG "metric": { 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG }, 11:39:14 DEBUG "pod": "ds-idrepo-0" 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:39:14 DEBUG "value": [ 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:39:14 DEBUG }, 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG "value": [ 11:39:14 DEBUG { 11:39:14 DEBUG 11:39:14 DEBUG { 11:39:14 DEBUG 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG 11:39:14 DEBUG "data": { 11:39:14 DEBUG [...] 11:39:14 DEBUG "data": { 11:39:14 DEBUG 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG [...] 11:39:14 DEBUG "result": [ 11:39:14 DEBUG 11:39:14 DEBUG "result": [ 11:39:14 DEBUG 11:39:14 DEBUG { 11:39:14 DEBUG 11:39:14 DEBUG { 11:39:14 DEBUG 11:39:14 DEBUG "metric": { 11:39:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:39:14 DEBUG "metric": { 11:39:14 DEBUG 11:39:14 DEBUG "instance": "10.142.0.11:9100" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG --- status code --- 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG "instance": "10.142.0.11:9100" 11:39:14 INFO [http_cmd]: http status code OK 11:39:14 DEBUG "pod": "ds-cts-0" 11:39:14 DEBUG }, 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG }, 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG }, 11:39:14 DEBUG --- status code --- 11:39:14 DEBUG }, 11:39:14 DEBUG "value": [ 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG "value": [ 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG "value": [ 11:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:39:14 DEBUG "value": [ 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG 11:39:14 DEBUG --- http response --- 11:39:14 DEBUG 11:39:14 DEBUG "0.0019989430984186734" 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:39:14 DEBUG "758149120" 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:39:14 DEBUG 11:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:39:14 DEBUG 11:39:14 DEBUG ] 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG ] 11:39:14 DEBUG { 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG 11:39:14 DEBUG ----- output ----- 11:39:14 DEBUG 11:39:14 DEBUG } 11:39:14 DEBUG { 11:39:14 DEBUG } 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG { 11:39:14 DEBUG [...] 11:39:14 DEBUG { 11:39:14 DEBUG [...] 11:39:14 DEBUG ] 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG ] 11:39:14 DEBUG "data": { 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG 11:39:14 DEBUG "status": "success", 11:39:14 DEBUG 11:39:14 DEBUG } 11:39:14 DEBUG "data": { 11:39:14 DEBUG } 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG "data": { 11:39:14 DEBUG 11:39:14 DEBUG "data": { 11:39:14 DEBUG 11:39:14 DEBUG } 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG } 11:39:14 DEBUG "result": [ 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG "result": [ 11:39:14 DEBUG { 11:39:14 DEBUG "metric": { 11:39:14 DEBUG "instance": "10.142.0.11:9100" 11:39:14 DEBUG { 11:39:14 DEBUG 11:39:14 DEBUG "resultType": "vector", 11:39:14 DEBUG 11:39:14 DEBUG "result": [ 11:39:14 DEBUG }, 11:39:14 DEBUG "metric": { 11:39:14 DEBUG "instance": "10.142.0.51:9100" 11:39:14 DEBUG "result": [ 11:39:14 DEBUG "instance": "10.142.0.51:9100" 11:39:14 DEBUG { 11:39:14 DEBUG "value": [ 11:39:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:39:14 DEBUG }, 11:39:14 DEBUG { 11:39:14 DEBUG }, 11:39:14 DEBUG "metric": { 11:39:14 DEBUG 11:39:14 DEBUG }, 11:39:14 DEBUG "value": [ 11:39:14 DEBUG "metric": { 11:39:14 DEBUG "value": [ 11:39:14 DEBUG "instance": "10.142.0.11:9100" 11:39:14 DEBUG 11:39:14 DEBUG "value": [ 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG }, 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG "0.00033333333333397756" 11:39:14 DEBUG }, 11:39:14 DEBUG "value": [ 11:39:14 DEBUG "value": [ 11:39:14 DEBUG [...] 11:39:14 DEBUG 11:39:14 DEBUG ] 11:39:14 DEBUG "1211686.1999999997" 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG } 11:39:14 DEBUG ] 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG [...] 11:39:14 DEBUG ] 11:39:14 DEBUG } 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG } 11:39:14 DEBUG ] 11:39:14 DEBUG [...] 11:39:14 DEBUG [...] 11:39:14 DEBUG "instance": "10.142.0.51:9100" 11:39:14 DEBUG 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG }, 11:39:14 DEBUG 11:39:14 DEBUG } 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG "value": [ 11:39:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:39:14 DEBUG 11:39:14 DEBUG 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG }, 11:39:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:39:14 DEBUG "instance": "10.142.0.51:9100" 11:39:14 DEBUG "1230972.9333333336" 11:39:14 DEBUG "value": [ 11:39:14 DEBUG }, 11:39:14 DEBUG }, 11:39:14 DEBUG ] 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG "value": [ 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG } 11:39:14 DEBUG "0.17376606383013687" 11:39:14 DEBUG "value": [ 11:39:14 DEBUG "14989955072" 11:39:14 DEBUG ] 11:39:14 DEBUG ] 11:39:14 DEBUG 1735904354, 11:39:14 DEBUG ] 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG "26077.866666666665" 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG ] 11:39:14 DEBUG ] 11:39:14 DEBUG ] 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG ] 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:14 DEBUG } 11:39:17 INFO 11:39:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:39:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:39:17 INFO [loop_until]: OK (rc = 0) 11:39:17 DEBUG --- stdout --- 11:39:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 170m 1% 3864Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 136m 0% 19055Mi 66% 11:39:17 DEBUG --- stderr --- 11:39:17 DEBUG 11:39:21 INFO 11:39:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:39:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:39:21 INFO [loop_until]: OK (rc = 0) 11:39:21 DEBUG --- stdout --- 11:39:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 74m 13478Mi rcs-645c86fdd6-j4rsk 67m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:39:21 DEBUG --- stderr --- 11:39:21 DEBUG 11:40:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904413" 11:40:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904413" 11:40:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904413" 11:40:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904413" 11:40:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904413" 11:40:13 INFO [http_cmd]: http status code OK 11:40:13 DEBUG --- status code --- 11:40:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:13 DEBUG --- http response --- 11:40:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:13 INFO [http_cmd]: http status code OK 11:40:13 DEBUG --- status code --- 11:40:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:13 DEBUG --- http response --- 11:40:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:13 ERROR [http_cmd]: ERROR 11:40:13 ERROR --- status code --- 11:40:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:40:13 ERROR --- http response --- 11:40:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:40:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:40:13 INFO [http_cmd]: http status code OK 11:40:13 DEBUG --- status code --- 11:40:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:13 DEBUG --- http response --- 11:40:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:13 INFO [http_cmd]: http status code OK 11:40:13 DEBUG --- status code --- 11:40:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:13 DEBUG --- http response --- 11:40:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904413" 11:40:13 INFO [http_cmd]: http status code OK 11:40:13 DEBUG --- status code --- 11:40:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:13 DEBUG --- http response --- 11:40:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- status code --- 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG { 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG "data": { 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG "result": [ 11:40:14 DEBUG { 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "pod": "ds-idrepo-0" 11:40:14 DEBUG }, 11:40:14 DEBUG "value": [ 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG [...] 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:40:14 DEBUG }, 11:40:14 DEBUG "value": [ 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG "749.885462918068" 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG } 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG { 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG "data": { 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG "result": [ 11:40:14 DEBUG { 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "pod": "ds-idrepo-0" 11:40:14 DEBUG }, 11:40:14 DEBUG "value": [ 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG [...] 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:40:14 DEBUG }, 11:40:14 DEBUG "value": [ 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG "0" 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG { 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG "data": { 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG "result": [ 11:40:14 DEBUG { 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "instance": "10.142.0.11:9100" 11:40:14 DEBUG }, 11:40:14 DEBUG "value": [ 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG [...] 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG "instance": "10.142.0.51:9100" 11:40:14 DEBUG }, 11:40:14 DEBUG "value": [ 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG "0.0003999999999981204" 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG } 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904414" 11:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904414" 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG --- http response --- 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG { 11:40:14 DEBUG { 11:40:14 DEBUG { 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:40:14 DEBUG "data": { 11:40:14 DEBUG "data": { 11:40:14 DEBUG "data": { 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG { 11:40:14 DEBUG "result": [ 11:40:14 DEBUG "result": [ 11:40:14 DEBUG "result": [ 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG { 11:40:14 DEBUG { 11:40:14 DEBUG { 11:40:14 DEBUG "data": { 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG "pod": "ds-idrepo-0" 11:40:14 DEBUG "instance": "10.142.0.11:9100" 11:40:14 DEBUG "instance": "10.142.0.11:9100" 11:40:14 DEBUG "result": [ 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG }, 11:40:14 DEBUG }, 11:40:14 DEBUG }, 11:40:14 DEBUG { 11:40:14 DEBUG --- http response --- 11:40:14 INFO [http_cmd]: http status code OK 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG "value": [ 11:40:14 DEBUG "value": [ 11:40:14 DEBUG "value": [ 11:40:14 DEBUG "metric": { 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:40:14 DEBUG --- status code --- 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG "pod": "ds-idrepo-0" 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG { 11:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG [...] 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG --- http response --- 11:40:14 DEBUG { 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG }, 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG 11:40:14 DEBUG "data": { 11:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG [...] 11:40:14 DEBUG [...] 11:40:14 DEBUG "value": [ 11:40:14 DEBUG { 11:40:14 DEBUG 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG ----- output ----- 11:40:14 DEBUG "data": { 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG 11:40:14 DEBUG "result": [ 11:40:14 DEBUG { 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG "data": { 11:40:14 DEBUG "instance": "10.142.0.51:9100" 11:40:14 DEBUG { 11:40:14 DEBUG "status": "success", 11:40:14 DEBUG "result": [ 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG }, 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "data": { 11:40:14 DEBUG { 11:40:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:40:14 DEBUG "instance": "10.142.0.51:9100" 11:40:14 DEBUG [...] 11:40:14 DEBUG "result": [ 11:40:14 DEBUG "value": [ 11:40:14 DEBUG "instance": "10.142.0.11:9100" 11:40:14 DEBUG }, 11:40:14 DEBUG "resultType": "vector", 11:40:14 DEBUG }, 11:40:14 DEBUG }, 11:40:14 DEBUG 11:40:14 DEBUG { 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "value": [ 11:40:14 DEBUG "result": [ 11:40:14 DEBUG "value": [ 11:40:14 DEBUG "value": [ 11:40:14 DEBUG 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "70644.09999999999" 11:40:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG { 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG 11:40:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:40:14 DEBUG ] 11:40:14 DEBUG }, 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG 11:40:14 DEBUG "metric": { 11:40:14 DEBUG "0.00033333333333397756" 11:40:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:40:14 DEBUG }, 11:40:14 DEBUG } 11:40:14 DEBUG "value": [ 11:40:14 DEBUG "758149120" 11:40:14 DEBUG [...] 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG }, 11:40:14 DEBUG ] 11:40:14 DEBUG 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG "instance": "10.142.0.51:9100" 11:40:14 DEBUG "value": [ 11:40:14 DEBUG } 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG [...] 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG }, 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG } 11:40:14 DEBUG "instance": "10.142.0.11:9100" 11:40:14 DEBUG ] 11:40:14 DEBUG "value": [ 11:40:14 DEBUG } 11:40:14 DEBUG 11:40:14 DEBUG "value": [ 11:40:14 DEBUG "0.0010498225008534227" 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG 11:40:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG }, 11:40:14 DEBUG } 11:40:14 DEBUG ] 11:40:14 DEBUG 11:40:14 DEBUG }, 11:40:14 DEBUG "89411.23333333332" 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG } 11:40:14 DEBUG 11:40:14 DEBUG "value": [ 11:40:14 DEBUG "value": [ 11:40:14 DEBUG ] 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG [...] 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG "14989434880" 11:40:14 DEBUG } 11:40:14 DEBUG } 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG } 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG } 11:40:14 DEBUG 11:40:14 DEBUG 11:40:14 DEBUG ] 11:40:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:40:14 DEBUG [...] 11:40:14 DEBUG } 11:40:14 DEBUG }, 11:40:14 DEBUG 11:40:14 DEBUG } 11:40:14 DEBUG "value": [ 11:40:14 DEBUG 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG 11:40:14 DEBUG "0.10319387327127726" 11:40:14 DEBUG "instance": "10.142.0.51:9100" 11:40:14 DEBUG ] 11:40:14 DEBUG }, 11:40:14 DEBUG } 11:40:14 DEBUG "value": [ 11:40:14 DEBUG ] 11:40:14 DEBUG 1735904414, 11:40:14 DEBUG } 11:40:14 DEBUG "26624" 11:40:14 DEBUG } 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG ] 11:40:14 DEBUG } 11:40:14 DEBUG } 11:40:17 INFO 11:40:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:40:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:40:17 INFO [loop_until]: OK (rc = 0) 11:40:17 DEBUG --- stdout --- 11:40:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 113m 0% 3865Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 78m 0% 19057Mi 67% 11:40:17 DEBUG --- stderr --- 11:40:17 DEBUG 11:40:21 INFO 11:40:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:40:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:40:21 INFO [loop_until]: OK (rc = 0) 11:40:21 DEBUG --- stdout --- 11:40:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:40:21 DEBUG --- stderr --- 11:40:21 DEBUG 11:41:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904473" 11:41:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904473" 11:41:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904473" 11:41:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904473" 11:41:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904473" 11:41:13 INFO [http_cmd]: http status code OK 11:41:13 DEBUG --- status code --- 11:41:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:13 DEBUG --- http response --- 11:41:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:13 INFO [http_cmd]: http status code OK 11:41:13 INFO [http_cmd]: http status code OK 11:41:13 INFO [http_cmd]: http status code OK 11:41:13 DEBUG --- status code --- 11:41:13 ERROR [http_cmd]: ERROR 11:41:13 DEBUG --- status code --- 11:41:13 DEBUG --- status code --- 11:41:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:13 ERROR --- status code --- 11:41:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:13 DEBUG --- http response --- 11:41:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:41:13 DEBUG --- http response --- 11:41:13 DEBUG --- http response --- 11:41:13 ERROR --- http response --- 11:41:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:41:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:41:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904473" 11:41:13 INFO [http_cmd]: http status code OK 11:41:13 DEBUG --- status code --- 11:41:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:13 DEBUG --- http response --- 11:41:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG { 11:41:14 DEBUG { 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG "data": { 11:41:14 DEBUG "data": { 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG "result": [ 11:41:14 DEBUG "result": [ 11:41:14 DEBUG { 11:41:14 DEBUG { 11:41:14 DEBUG "metric": { 11:41:14 DEBUG "metric": { 11:41:14 DEBUG "pod": "ds-idrepo-0" 11:41:14 DEBUG "pod": "ds-idrepo-0" 11:41:14 DEBUG }, 11:41:14 DEBUG }, 11:41:14 DEBUG "value": [ 11:41:14 DEBUG "value": [ 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG [...] 11:41:14 DEBUG [...] 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG "pod": "ds-cts-0" 11:41:14 DEBUG "pod": "ds-cts-0" 11:41:14 DEBUG }, 11:41:14 DEBUG }, 11:41:14 DEBUG "value": [ 11:41:14 DEBUG "value": [ 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG "0" 11:41:14 DEBUG "584.9757212225079" 11:41:14 DEBUG ] 11:41:14 DEBUG ] 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG ] 11:41:14 DEBUG ] 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG { 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG "data": { 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG "result": [ 11:41:14 DEBUG { 11:41:14 DEBUG "metric": { 11:41:14 DEBUG "instance": "10.142.0.11:9100" 11:41:14 DEBUG }, 11:41:14 DEBUG "value": [ 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG [...] 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG "instance": "10.142.0.51:9100" 11:41:14 DEBUG }, 11:41:14 DEBUG "value": [ 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG "0.0003999999999981204" 11:41:14 DEBUG ] 11:41:14 DEBUG } 11:41:14 DEBUG ] 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904474" 11:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904474" 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG ----- output ----- 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG { 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG "data": { 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:41:14 DEBUG "resultType": "vector", 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:41:14 DEBUG "result": [ 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG { 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG { 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG { 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG { 11:41:14 DEBUG "metric": { 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG "data": { 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG "pod": "ds-idrepo-0" 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:41:14 DEBUG "data": { 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG "data": { 11:41:14 DEBUG }, 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG "result": [ 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG "value": [ 11:41:14 DEBUG { 11:41:14 DEBUG "result": [ 11:41:14 DEBUG { 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG "result": [ 11:41:14 DEBUG 11:41:14 INFO [http_cmd]: http status code OK 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG { 11:41:14 DEBUG "metric": { 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG { 11:41:14 DEBUG 11:41:14 DEBUG --- status code --- 11:41:14 DEBUG "data": { 11:41:14 DEBUG "metric": { 11:41:14 DEBUG "instance": "10.142.0.11:9100" 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG "metric": { 11:41:14 DEBUG 11:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG "instance": "10.142.0.11:9100" 11:41:14 DEBUG }, 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG "instance": "10.142.0.11:9100" 11:41:14 DEBUG [...] 11:41:14 DEBUG --- http response --- 11:41:14 DEBUG "result": [ 11:41:14 DEBUG }, 11:41:14 DEBUG "value": [ 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:41:14 DEBUG }, 11:41:14 DEBUG 11:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:41:14 DEBUG { 11:41:14 DEBUG "value": [ 11:41:14 DEBUG 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG "value": [ 11:41:14 DEBUG 11:41:14 DEBUG ----- output ----- 11:41:14 DEBUG "metric": { 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG { 11:41:14 DEBUG { 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG { 11:41:14 DEBUG "pod": "ds-idrepo-0" 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG 11:41:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:41:14 DEBUG "status": "success", 11:41:14 DEBUG }, 11:41:14 DEBUG 11:41:14 DEBUG [...] 11:41:14 DEBUG "data": { 11:41:14 DEBUG "data": { 11:41:14 DEBUG 11:41:14 DEBUG }, 11:41:14 DEBUG "data": { 11:41:14 DEBUG "value": [ 11:41:14 DEBUG [...] 11:41:14 DEBUG 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG [...] 11:41:14 DEBUG "value": [ 11:41:14 DEBUG "resultType": "vector", 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG "result": [ 11:41:14 DEBUG "result": [ 11:41:14 DEBUG 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG "result": [ 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG { 11:41:14 DEBUG { 11:41:14 DEBUG 11:41:14 DEBUG "758149120" 11:41:14 DEBUG { 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG "instance": "10.142.0.51:9100" 11:41:14 DEBUG "metric": { 11:41:14 DEBUG "metric": { 11:41:14 DEBUG 11:41:14 DEBUG ] 11:41:14 DEBUG "metric": { 11:41:14 DEBUG [...] 11:41:14 DEBUG "instance": "10.142.0.51:9100" 11:41:14 DEBUG }, 11:41:14 DEBUG "instance": "10.142.0.11:9100" 11:41:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:41:14 DEBUG "instance": "10.142.0.51:9100" 11:41:14 DEBUG } 11:41:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:41:14 DEBUG 11:41:14 DEBUG }, 11:41:14 DEBUG "value": [ 11:41:14 DEBUG }, 11:41:14 DEBUG }, 11:41:14 DEBUG }, 11:41:14 DEBUG ] 11:41:14 DEBUG }, 11:41:14 DEBUG 11:41:14 DEBUG "value": [ 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG "value": [ 11:41:14 DEBUG "value": [ 11:41:14 DEBUG "value": [ 11:41:14 DEBUG } 11:41:14 DEBUG "value": [ 11:41:14 DEBUG 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG "0.000666666666666534" 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG } 11:41:14 DEBUG 11:41:14 DEBUG "pod": "ds-cts-0" 11:41:14 DEBUG "72080.49999999999" 11:41:14 DEBUG ] 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG "90700.60000000002" 11:41:14 DEBUG 11:41:14 DEBUG }, 11:41:14 DEBUG ] 11:41:14 DEBUG } 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG ] 11:41:14 DEBUG 11:41:14 DEBUG "value": [ 11:41:14 DEBUG } 11:41:14 DEBUG ] 11:41:14 DEBUG [...] 11:41:14 DEBUG [...] 11:41:14 DEBUG } 11:41:14 DEBUG [...] 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG ] 11:41:14 DEBUG } 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG ] 11:41:14 DEBUG 11:41:14 DEBUG "0.0027182947729188405" 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG } 11:41:14 DEBUG 11:41:14 DEBUG ] 11:41:14 DEBUG } 11:41:14 DEBUG 11:41:14 DEBUG 11:41:14 DEBUG } 11:41:14 DEBUG 11:41:14 DEBUG } 11:41:14 DEBUG "instance": "10.142.0.51:9100" 11:41:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:41:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:41:14 DEBUG ] 11:41:14 DEBUG }, 11:41:14 DEBUG }, 11:41:14 DEBUG }, 11:41:14 DEBUG } 11:41:14 DEBUG "value": [ 11:41:14 DEBUG "value": [ 11:41:14 DEBUG "value": [ 11:41:14 DEBUG } 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG 1735904474, 11:41:14 DEBUG "14990139392" 11:41:14 DEBUG "0.731898797803493" 11:41:14 DEBUG "20343.466666666667" 11:41:14 DEBUG ] 11:41:14 DEBUG ] 11:41:14 DEBUG ] 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG ] 11:41:14 DEBUG ] 11:41:14 DEBUG ] 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:14 DEBUG } 11:41:17 INFO 11:41:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:41:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:41:17 INFO [loop_until]: OK (rc = 0) 11:41:17 DEBUG --- stdout --- 11:41:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 112m 0% 3836Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 76m 0% 19057Mi 67% 11:41:17 DEBUG --- stderr --- 11:41:17 DEBUG 11:41:21 INFO 11:41:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:41:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:41:21 INFO [loop_until]: OK (rc = 0) 11:41:21 DEBUG --- stdout --- 11:41:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:41:21 DEBUG --- stderr --- 11:41:21 DEBUG 11:42:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904533" 11:42:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904533" 11:42:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904533" 11:42:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904533" 11:42:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904533" 11:42:13 INFO [http_cmd]: http status code OK 11:42:13 DEBUG --- status code --- 11:42:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:13 DEBUG --- http response --- 11:42:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:13 INFO [http_cmd]: http status code OK 11:42:13 DEBUG --- status code --- 11:42:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:13 DEBUG --- http response --- 11:42:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:13 INFO [http_cmd]: http status code OK 11:42:13 DEBUG --- status code --- 11:42:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:13 DEBUG --- http response --- 11:42:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:13 ERROR [http_cmd]: ERROR 11:42:13 ERROR --- status code --- 11:42:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:42:13 ERROR --- http response --- 11:42:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:42:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:42:13 INFO [http_cmd]: http status code OK 11:42:13 DEBUG --- status code --- 11:42:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:13 DEBUG --- http response --- 11:42:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904533" 11:42:13 INFO [http_cmd]: http status code OK 11:42:13 DEBUG --- status code --- 11:42:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:13 DEBUG --- http response --- 11:42:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 DEBUG --- http response --- 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG --- status code --- 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG --- http response --- 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735904534, "0" ] } ] } } 11:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735904534, "651.5238277024877" ] } ] } } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:42:14 DEBUG ----- output ----- 11:42:14 DEBUG { 11:42:14 DEBUG "status": "success", 11:42:14 DEBUG "data": { 11:42:14 DEBUG "resultType": "vector", 11:42:14 DEBUG "result": [ 11:42:14 DEBUG { 11:42:14 DEBUG "metric": { 11:42:14 DEBUG "instance": "10.142.0.11:9100" 11:42:14 DEBUG }, 11:42:14 DEBUG "value": [ 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG [...] 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG "instance": "10.142.0.51:9100" 11:42:14 DEBUG }, 11:42:14 DEBUG "value": [ 11:42:14 DEBUG 1735904534, 11:42:14 DEBUG "0.0011999999999981507" 11:42:14 DEBUG ] 11:42:14 DEBUG } 11:42:14 DEBUG ] 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904534" 11:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904534" 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:42:14 DEBUG ----- output ----- 11:42:14 DEBUG { 11:42:14 DEBUG "status": "success", 11:42:14 DEBUG "data": { 11:42:14 DEBUG "resultType": "vector", 11:42:14 DEBUG "result": [ 11:42:14 DEBUG { 11:42:14 DEBUG "metric": { 11:42:14 DEBUG "pod": "ds-idrepo-0" 11:42:14 DEBUG }, 11:42:14 DEBUG "value": [ 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG [...] 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:42:14 DEBUG }, 11:42:14 DEBUG "value": [ 11:42:14 DEBUG 1735904534, 11:42:14 DEBUG "758149120" 11:42:14 DEBUG ] 11:42:14 DEBUG } 11:42:14 DEBUG ] 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:42:14 DEBUG ----- output ----- 11:42:14 DEBUG { 11:42:14 DEBUG "status": "success", 11:42:14 DEBUG "data": { 11:42:14 DEBUG "resultType": "vector", 11:42:14 DEBUG "result": [ 11:42:14 DEBUG { 11:42:14 DEBUG "metric": { 11:42:14 DEBUG "instance": "10.142.0.11:9100" 11:42:14 DEBUG }, 11:42:14 DEBUG "value": [ 11:42:14 DEBUG 11:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG 11:42:14 DEBUG ----- output ----- 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG 11:42:14 DEBUG { 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG [...] 11:42:14 DEBUG "status": "success", 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG 11:42:14 DEBUG "data": { 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG 11:42:14 DEBUG "resultType": "vector", 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG ----- output ----- 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG 11:42:14 DEBUG "result": [ 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG { 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG "instance": "10.142.0.51:9100" 11:42:14 DEBUG { 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG "status": "success", 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG }, 11:42:14 DEBUG "metric": { 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:42:14 DEBUG "data": { 11:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:42:14 DEBUG "value": [ 11:42:14 DEBUG "instance": "10.142.0.11:9100" 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG ----- output ----- 11:42:14 DEBUG "resultType": "vector", 11:42:14 DEBUG ----- output ----- 11:42:14 INFO [http_cmd]: http status code OK 11:42:14 DEBUG 1735904534, 11:42:14 DEBUG }, 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG { 11:42:14 DEBUG "result": [ 11:42:14 DEBUG { 11:42:14 DEBUG --- status code --- 11:42:14 DEBUG "0.0009999999999995642" 11:42:14 DEBUG "value": [ 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG "status": "success", 11:42:14 DEBUG { 11:42:14 DEBUG "status": "success", 11:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:42:14 DEBUG ] 11:42:14 DEBUG 11:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:42:14 DEBUG "data": { 11:42:14 DEBUG "metric": { 11:42:14 DEBUG "data": { 11:42:14 DEBUG --- http response --- 11:42:14 DEBUG } 11:42:14 DEBUG 11:42:14 DEBUG ----- output ----- 11:42:14 DEBUG "resultType": "vector", 11:42:14 DEBUG "pod": "ds-cts-0" 11:42:14 DEBUG "resultType": "vector", 11:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:42:14 DEBUG ] 11:42:14 DEBUG 11:42:14 DEBUG { 11:42:14 DEBUG "result": [ 11:42:14 DEBUG }, 11:42:14 DEBUG "result": [ 11:42:14 DEBUG ----- output ----- 11:42:14 DEBUG } 11:42:14 DEBUG [...] 11:42:14 DEBUG "status": "success", 11:42:14 DEBUG { 11:42:14 DEBUG "value": [ 11:42:14 DEBUG { 11:42:14 DEBUG { 11:42:14 DEBUG } 11:42:14 DEBUG 11:42:14 DEBUG "data": { 11:42:14 DEBUG "metric": { 11:42:14 DEBUG 11:42:14 DEBUG "metric": { 11:42:14 DEBUG "status": "success", 11:42:14 DEBUG 11:42:14 DEBUG "resultType": "vector", 11:42:14 DEBUG "instance": "10.142.0.11:9100" 11:42:14 DEBUG 11:42:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:42:14 DEBUG "data": { 11:42:14 DEBUG 11:42:14 DEBUG "result": [ 11:42:14 DEBUG }, 11:42:14 DEBUG 11:42:14 DEBUG }, 11:42:14 DEBUG "resultType": "vector", 11:42:14 DEBUG "instance": "10.142.0.51:9100" 11:42:14 DEBUG { 11:42:14 DEBUG "value": [ 11:42:14 DEBUG [...] 11:42:14 DEBUG "value": [ 11:42:14 DEBUG "result": [ 11:42:14 DEBUG }, 11:42:14 DEBUG "metric": { 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG { 11:42:14 DEBUG "value": [ 11:42:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG "metric": { 11:42:14 DEBUG 1735904534, 11:42:14 DEBUG }, 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG "instance": "10.142.0.11:9100" 11:42:14 DEBUG "89269.03333333335" 11:42:14 DEBUG "value": [ 11:42:14 DEBUG [...] 11:42:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:42:14 DEBUG [...] 11:42:14 DEBUG }, 11:42:14 DEBUG ] 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG }, 11:42:14 DEBUG 11:42:14 DEBUG "value": [ 11:42:14 DEBUG } 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG "value": [ 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG ] 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG 1735904534, 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG } 11:42:14 DEBUG [...] 11:42:14 DEBUG "instance": "10.142.0.51:9100" 11:42:14 DEBUG "0" 11:42:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:42:14 DEBUG 11:42:14 DEBUG } 11:42:14 DEBUG 11:42:14 DEBUG }, 11:42:14 DEBUG ] 11:42:14 DEBUG }, 11:42:14 DEBUG [...] 11:42:14 DEBUG 11:42:14 DEBUG "value": [ 11:42:14 DEBUG } 11:42:14 DEBUG "value": [ 11:42:14 DEBUG 11:42:14 DEBUG 11:42:14 DEBUG 1735904534, 11:42:14 DEBUG ] 11:42:14 DEBUG 1735904534, 11:42:14 DEBUG 11:42:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:42:14 DEBUG }, 11:42:14 DEBUG "70475.30000000002" 11:42:14 DEBUG "14991290368" 11:42:14 DEBUG 11:42:14 DEBUG "value": [ 11:42:14 DEBUG } 11:42:14 DEBUG ] 11:42:14 DEBUG ] 11:42:14 DEBUG "instance": "10.142.0.51:9100" 11:42:14 DEBUG 1735904534, 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:14 DEBUG }, 11:42:14 DEBUG "0.09015412980666362" 11:42:14 DEBUG ] 11:42:14 DEBUG } 11:42:14 DEBUG "value": [ 11:42:14 DEBUG 1735904534, 11:42:14 DEBUG "26760.533333333333" 11:42:14 DEBUG ] 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:14 DEBUG ] 11:42:14 DEBUG } 11:42:14 DEBUG ] 11:42:14 DEBUG ] 11:42:14 DEBUG ] 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:14 DEBUG } 11:42:17 INFO 11:42:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:42:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:42:17 INFO [loop_until]: OK (rc = 0) 11:42:17 DEBUG --- stdout --- 11:42:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 110m 0% 3838Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 83m 0% 19051Mi 66% 11:42:17 DEBUG --- stderr --- 11:42:17 DEBUG 11:42:21 INFO 11:42:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:42:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:42:21 INFO [loop_until]: OK (rc = 0) 11:42:21 DEBUG --- stdout --- 11:42:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 46m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:42:21 DEBUG --- stderr --- 11:42:21 DEBUG 11:43:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904593" 11:43:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904593" 11:43:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904593" 11:43:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904593" 11:43:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904593" 11:43:13 INFO [http_cmd]: http status code OK 11:43:13 DEBUG --- status code --- 11:43:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:13 DEBUG --- http response --- 11:43:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:13 INFO [http_cmd]: http status code OK 11:43:13 INFO [http_cmd]: http status code OK 11:43:13 DEBUG --- status code --- 11:43:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:13 INFO [http_cmd]: http status code OK 11:43:13 DEBUG --- http response --- 11:43:13 DEBUG --- status code --- 11:43:13 DEBUG --- status code --- 11:43:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:13 DEBUG --- http response --- 11:43:13 DEBUG --- http response --- 11:43:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:13 ERROR [http_cmd]: ERROR 11:43:13 ERROR --- status code --- 11:43:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:43:13 ERROR --- http response --- 11:43:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:43:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:43:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904593" 11:43:13 INFO [http_cmd]: http status code OK 11:43:13 DEBUG --- status code --- 11:43:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:13 DEBUG --- http response --- 11:43:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- status code --- 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG --- http response --- 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904594" 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG { 11:43:14 DEBUG { 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG "data": { 11:43:14 DEBUG "data": { 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG "result": [ 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG { 11:43:14 DEBUG "result": [ 11:43:14 DEBUG "metric": { 11:43:14 DEBUG { 11:43:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:43:14 DEBUG "metric": { 11:43:14 DEBUG }, 11:43:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:43:14 DEBUG "value": [ 11:43:14 DEBUG }, 11:43:14 DEBUG 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG [...] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG [...] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG "pod": "ds-cts-0" 11:43:14 DEBUG 11:43:14 DEBUG }, 11:43:14 DEBUG "pod": "ds-cts-0" 11:43:14 DEBUG "value": [ 11:43:14 DEBUG }, 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG "0" 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG ] 11:43:14 DEBUG "522.0176108749843" 11:43:14 DEBUG } 11:43:14 DEBUG ] 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG { 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG "data": { 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG "result": [ 11:43:14 DEBUG { 11:43:14 DEBUG "metric": { 11:43:14 DEBUG "instance": "10.142.0.11:9100" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG [...] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG "instance": "10.142.0.51:9100" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG "0.00046666666666794754" 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG { 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG "data": { 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG "result": [ 11:43:14 DEBUG { 11:43:14 DEBUG "metric": { 11:43:14 DEBUG "pod": "ds-idrepo-0" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG [...] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG "758149120" 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904594" 11:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904594" 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG { 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG "data": { 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG "result": [ 11:43:14 DEBUG { 11:43:14 DEBUG "metric": { 11:43:14 DEBUG "instance": "10.142.0.11:9100" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG [...] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG "instance": "10.142.0.51:9100" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG "0.0006666666666660604" 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG { 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG "data": { 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG "result": [ 11:43:14 DEBUG { 11:43:14 DEBUG "metric": { 11:43:14 DEBUG "instance": "10.142.0.11:9100" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG [...] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG "instance": "10.142.0.51:9100" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG "792895.3666666668" 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG { 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG "data": { 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG "result": [ 11:43:14 DEBUG { 11:43:14 DEBUG "metric": { 11:43:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG [...] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG "pod": "ds-cts-0" 11:43:14 DEBUG }, 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG "0.0016536080324936744" 11:43:14 DEBUG ] 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG } 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG ] 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG } 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG } 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG { 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG "data": { 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG "result": [ 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG { 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG "metric": { 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:43:14 INFO [http_cmd]: http status code OK 11:43:14 DEBUG "instance": "10.142.0.11:9100" 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG --- status code --- 11:43:14 DEBUG }, 11:43:14 DEBUG { 11:43:14 DEBUG { 11:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:43:14 DEBUG "value": [ 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG --- http response --- 11:43:14 DEBUG 11:43:14 DEBUG "data": { 11:43:14 DEBUG "data": { 11:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:43:14 DEBUG 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG ----- output ----- 11:43:14 DEBUG 11:43:14 DEBUG "result": [ 11:43:14 DEBUG "result": [ 11:43:14 DEBUG { 11:43:14 DEBUG [...] 11:43:14 DEBUG { 11:43:14 DEBUG { 11:43:14 DEBUG "status": "success", 11:43:14 DEBUG 11:43:14 DEBUG "metric": { 11:43:14 DEBUG "metric": { 11:43:14 DEBUG "data": { 11:43:14 DEBUG "resultType": "vector", 11:43:14 DEBUG "result": [ 11:43:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:43:14 DEBUG 11:43:14 DEBUG "instance": "10.142.0.11:9100" 11:43:14 DEBUG { 11:43:14 DEBUG }, 11:43:14 DEBUG 11:43:14 DEBUG }, 11:43:14 DEBUG "metric": { 11:43:14 DEBUG "value": [ 11:43:14 DEBUG "instance": "10.142.0.51:9100" 11:43:14 DEBUG "value": [ 11:43:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:43:14 DEBUG 11:43:14 DEBUG }, 11:43:14 DEBUG 11:43:14 DEBUG }, 11:43:14 DEBUG 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 11:43:14 DEBUG "value": [ 11:43:14 DEBUG 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG [...] 11:43:14 DEBUG "26214.399999999998" 11:43:14 DEBUG [...] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG ] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG } 11:43:14 DEBUG 11:43:14 DEBUG [...] 11:43:14 DEBUG 11:43:14 DEBUG ] 11:43:14 DEBUG 11:43:14 DEBUG 11:43:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:43:14 DEBUG } 11:43:14 DEBUG "instance": "10.142.0.51:9100" 11:43:14 DEBUG 11:43:14 DEBUG }, 11:43:14 DEBUG } 11:43:14 DEBUG }, 11:43:14 DEBUG 11:43:14 DEBUG "value": [ 11:43:14 DEBUG "value": [ 11:43:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG }, 11:43:14 DEBUG "0.08673206643345467" 11:43:14 DEBUG "775327.7000000002" 11:43:14 DEBUG "value": [ 11:43:14 DEBUG ] 11:43:14 DEBUG ] 11:43:14 DEBUG 1735904594, 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 DEBUG "14986969088" 11:43:14 DEBUG ] 11:43:14 DEBUG ] 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:14 DEBUG ] 11:43:14 DEBUG } 11:43:14 DEBUG } 11:43:17 INFO 11:43:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:43:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:43:17 INFO [loop_until]: OK (rc = 0) 11:43:17 DEBUG --- stdout --- 11:43:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 141m 0% 3872Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 97m 0% 19055Mi 67% 11:43:17 DEBUG --- stderr --- 11:43:17 DEBUG 11:43:21 INFO 11:43:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:43:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:43:21 INFO [loop_until]: OK (rc = 0) 11:43:21 DEBUG --- stdout --- 11:43:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 47m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:43:21 DEBUG --- stderr --- 11:43:21 DEBUG 11:44:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904653" 11:44:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904653" 11:44:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904653" 11:44:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904653" 11:44:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904653" 11:44:13 INFO [http_cmd]: http status code OK 11:44:13 DEBUG --- status code --- 11:44:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:13 DEBUG --- http response --- 11:44:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:13 INFO [http_cmd]: http status code OK 11:44:13 DEBUG --- status code --- 11:44:13 ERROR [http_cmd]: ERROR 11:44:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:13 ERROR --- status code --- 11:44:13 DEBUG --- http response --- 11:44:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:44:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:13 ERROR --- http response --- 11:44:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:44:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:44:13 INFO [http_cmd]: http status code OK 11:44:13 DEBUG --- status code --- 11:44:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:13 DEBUG --- http response --- 11:44:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:13 INFO [http_cmd]: http status code OK 11:44:13 DEBUG --- status code --- 11:44:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:13 DEBUG --- http response --- 11:44:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904653" 11:44:13 INFO [http_cmd]: http status code OK 11:44:13 DEBUG --- status code --- 11:44:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:13 DEBUG --- http response --- 11:44:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904654" 11:44:14 DEBUG --- status code --- 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:44:14 DEBUG { 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:44:14 DEBUG { 11:44:14 DEBUG "data": { 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG { 11:44:14 DEBUG "data": { 11:44:14 DEBUG "result": [ 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG { 11:44:14 DEBUG "data": { 11:44:14 DEBUG "result": [ 11:44:14 DEBUG "metric": { 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG { 11:44:14 DEBUG "pod": "ds-cts-0" 11:44:14 DEBUG "result": [ 11:44:14 DEBUG "metric": { 11:44:14 DEBUG }, 11:44:14 DEBUG { 11:44:14 DEBUG "pod": "ds-cts-0" 11:44:14 DEBUG "value": [ 11:44:14 DEBUG "metric": { 11:44:14 DEBUG }, 11:44:14 DEBUG 11:44:14 DEBUG "instance": "10.142.0.11:9100" 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG }, 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG [...] 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG [...] 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG [...] 11:44:14 DEBUG 11:44:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG }, 11:44:14 DEBUG 11:44:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG }, 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG "instance": "10.142.0.51:9100" 11:44:14 DEBUG "value": [ 11:44:14 DEBUG "221.53469137693367" 11:44:14 DEBUG }, 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG ] 11:44:14 DEBUG "value": [ 11:44:14 DEBUG "0" 11:44:14 DEBUG } 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG ] 11:44:14 DEBUG ] 11:44:14 DEBUG "0.00043333333333066547" 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 DEBUG ] 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG { 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG "data": { 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG "result": [ 11:44:14 DEBUG { 11:44:14 DEBUG "metric": { 11:44:14 DEBUG "pod": "ds-idrepo-0" 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG [...] 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG "758149120" 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG { 11:44:14 DEBUG { 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG "data": { 11:44:14 DEBUG "data": { 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG "result": [ 11:44:14 DEBUG "result": [ 11:44:14 DEBUG { 11:44:14 DEBUG { 11:44:14 DEBUG "metric": { 11:44:14 DEBUG "metric": { 11:44:14 DEBUG "instance": "10.142.0.11:9100" 11:44:14 DEBUG "instance": "10.142.0.11:9100" 11:44:14 DEBUG }, 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG [...] 11:44:14 DEBUG [...] 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG "instance": "10.142.0.51:9100" 11:44:14 DEBUG "instance": "10.142.0.51:9100" 11:44:14 DEBUG }, 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG "1229829.8666666667" 11:44:14 DEBUG "0.0003333333333330302" 11:44:14 DEBUG ] 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904654" 11:44:14 DEBUG ] 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904654" 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904654" 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904654" 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG { 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG "data": { 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG "result": [ 11:44:14 DEBUG { 11:44:14 DEBUG "metric": { 11:44:14 DEBUG "pod": "ds-cts-0" 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG [...] 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG "0.0003997468027269785" 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG { 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG "data": { 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG "result": [ 11:44:14 DEBUG { 11:44:14 DEBUG "metric": { 11:44:14 DEBUG "instance": "10.142.0.11:9100" 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG [...] 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG "instance": "10.142.0.51:9100" 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG "28262.4" 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG { 11:44:14 DEBUG { 11:44:14 DEBUG "status": "success", 11:44:14 INFO [http_cmd]: http status code OK 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG "data": { 11:44:14 DEBUG --- status code --- 11:44:14 DEBUG "data": { 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG "result": [ 11:44:14 DEBUG --- http response --- 11:44:14 DEBUG "result": [ 11:44:14 DEBUG { 11:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:44:14 DEBUG { 11:44:14 DEBUG "metric": { 11:44:14 DEBUG ----- output ----- 11:44:14 DEBUG "metric": { 11:44:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:44:14 DEBUG { 11:44:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:44:14 DEBUG }, 11:44:14 DEBUG "status": "success", 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG "data": { 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG "resultType": "vector", 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG "result": [ 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG { 11:44:14 DEBUG 11:44:14 DEBUG [...] 11:44:14 DEBUG "metric": { 11:44:14 DEBUG [...] 11:44:14 DEBUG 11:44:14 DEBUG "instance": "10.142.0.11:9100" 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG }, 11:44:14 DEBUG 11:44:14 DEBUG 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:44:14 DEBUG 11:44:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:44:14 DEBUG }, 11:44:14 DEBUG 11:44:14 DEBUG }, 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG [...] 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG "14989942784" 11:44:14 DEBUG 11:44:14 DEBUG "0.04014780666485751" 11:44:14 DEBUG ] 11:44:14 DEBUG 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG 11:44:14 DEBUG } 11:44:14 DEBUG ] 11:44:14 DEBUG "instance": "10.142.0.51:9100" 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG }, 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:14 DEBUG "value": [ 11:44:14 DEBUG 1735904654, 11:44:14 DEBUG "1210567.866666667" 11:44:14 DEBUG } 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG ] 11:44:14 DEBUG } 11:44:14 DEBUG } 11:44:17 INFO 11:44:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:44:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:44:17 INFO [loop_until]: OK (rc = 0) 11:44:17 DEBUG --- stdout --- 11:44:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 171m 1% 3888Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 177m 0% 19055Mi 67% 11:44:17 DEBUG --- stderr --- 11:44:17 DEBUG 11:44:21 INFO 11:44:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:44:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:44:21 INFO [loop_until]: OK (rc = 0) 11:44:21 DEBUG --- stdout --- 11:44:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 92m 13478Mi rcs-645c86fdd6-j4rsk 49m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:44:21 DEBUG --- stderr --- 11:44:21 DEBUG 11:45:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904713" 11:45:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904713" 11:45:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904713" 11:45:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904713" 11:45:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904713" 11:45:13 INFO [http_cmd]: http status code OK 11:45:13 DEBUG --- status code --- 11:45:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:13 DEBUG --- http response --- 11:45:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:13 INFO [http_cmd]: http status code OK 11:45:13 INFO [http_cmd]: http status code OK 11:45:13 ERROR [http_cmd]: ERROR 11:45:13 DEBUG --- status code --- 11:45:13 DEBUG --- status code --- 11:45:13 ERROR --- status code --- 11:45:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:45:13 DEBUG --- http response --- 11:45:13 DEBUG --- http response --- 11:45:13 ERROR --- http response --- 11:45:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:45:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:45:13 INFO [http_cmd]: http status code OK 11:45:13 DEBUG --- status code --- 11:45:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:13 DEBUG --- http response --- 11:45:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904713" 11:45:13 INFO [http_cmd]: http status code OK 11:45:13 DEBUG --- status code --- 11:45:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:13 DEBUG --- http response --- 11:45:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG --- status code --- 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG { 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG { 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG { 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG "data": { 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG "data": { 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG "data": { 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG "result": [ 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG "result": [ 11:45:14 DEBUG { 11:45:14 DEBUG "result": [ 11:45:14 DEBUG { 11:45:14 DEBUG "metric": { 11:45:14 DEBUG { 11:45:14 DEBUG "metric": { 11:45:14 DEBUG "pod": "ds-cts-0" 11:45:14 DEBUG "metric": { 11:45:14 DEBUG "pod": "ds-cts-0" 11:45:14 DEBUG }, 11:45:14 DEBUG "instance": "10.142.0.11:9100" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:45:14 DEBUG 11:45:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:45:14 DEBUG }, 11:45:14 DEBUG "instance": "10.142.0.51:9100" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG "0" 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG "868.9558598494207" 11:45:14 DEBUG ] 11:45:14 DEBUG "0.0005333333333320903" 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG { 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG "data": { 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG "result": [ 11:45:14 DEBUG { 11:45:14 DEBUG "metric": { 11:45:14 DEBUG "pod": "ds-idrepo-0" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG "758149120" 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904714" 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG { 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG "data": { 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG "result": [ 11:45:14 DEBUG { 11:45:14 DEBUG "metric": { 11:45:14 DEBUG "instance": "10.142.0.11:9100" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG "instance": "10.142.0.51:9100" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG "1239217.4333333333" 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG { 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG "data": { 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG "result": [ 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG { 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG "metric": { 11:45:14 DEBUG { 11:45:14 DEBUG "instance": "10.142.0.11:9100" 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG }, 11:45:14 DEBUG "data": { 11:45:14 DEBUG "value": [ 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG 11:45:14 DEBUG "result": [ 11:45:14 DEBUG 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904714" 11:45:14 DEBUG { 11:45:14 DEBUG 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904714" 11:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904714" 11:45:14 DEBUG "metric": { 11:45:14 DEBUG [...] 11:45:14 DEBUG "pod": "ds-cts-0" 11:45:14 DEBUG 11:45:14 DEBUG }, 11:45:14 DEBUG 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG "instance": "10.142.0.51:9100" 11:45:14 DEBUG 11:45:14 DEBUG }, 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG "0" 11:45:14 DEBUG 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG ] 11:45:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG } 11:45:14 DEBUG }, 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG ] 11:45:14 DEBUG "value": [ 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG } 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG { 11:45:14 DEBUG } 11:45:14 DEBUG "0.0011234091647649943" 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG ] 11:45:14 DEBUG "data": { 11:45:14 DEBUG } 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG ] 11:45:14 DEBUG "result": [ 11:45:14 DEBUG } 11:45:14 DEBUG { 11:45:14 DEBUG } 11:45:14 DEBUG "metric": { 11:45:14 DEBUG "instance": "10.142.0.11:9100" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG "instance": "10.142.0.51:9100" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG "34816" 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 INFO [http_cmd]: http status code OK 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- status code --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG --- http response --- 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG { 11:45:14 DEBUG ----- output ----- 11:45:14 DEBUG { 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG { 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG "data": { 11:45:14 DEBUG "status": "success", 11:45:14 DEBUG "data": { 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG "data": { 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG "result": [ 11:45:14 DEBUG "resultType": "vector", 11:45:14 DEBUG "result": [ 11:45:14 DEBUG { 11:45:14 DEBUG "result": [ 11:45:14 DEBUG { 11:45:14 DEBUG "metric": { 11:45:14 DEBUG { 11:45:14 DEBUG "metric": { 11:45:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:45:14 DEBUG "metric": { 11:45:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:45:14 DEBUG }, 11:45:14 DEBUG "instance": "10.142.0.11:9100" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG 11:45:14 DEBUG [...] 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG 11:45:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:45:14 DEBUG 11:45:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:45:14 DEBUG }, 11:45:14 DEBUG "instance": "10.142.0.51:9100" 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG }, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG "value": [ 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG "14989443072" 11:45:14 DEBUG 1735904714, 11:45:14 DEBUG "0.0334144448217806" 11:45:14 DEBUG ] 11:45:14 DEBUG "1220454.5000000005" 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG ] 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:14 DEBUG } 11:45:17 INFO 11:45:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:45:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:45:17 INFO [loop_until]: OK (rc = 0) 11:45:17 DEBUG --- stdout --- 11:45:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 164m 1% 3885Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 130m 0% 19058Mi 67% 11:45:17 DEBUG --- stderr --- 11:45:17 DEBUG 11:45:21 INFO 11:45:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:45:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:45:21 INFO [loop_until]: OK (rc = 0) 11:45:21 DEBUG --- stdout --- 11:45:21 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 64m 13478Mi rcs-645c86fdd6-j4rsk 66m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:45:21 DEBUG --- stderr --- 11:45:21 DEBUG 11:46:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904773" 11:46:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904773" 11:46:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904773" 11:46:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904773" 11:46:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904773" 11:46:13 ERROR [http_cmd]: ERROR 11:46:13 INFO [http_cmd]: http status code OK 11:46:13 ERROR --- status code --- 11:46:13 DEBUG --- status code --- 11:46:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:46:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:13 ERROR --- http response --- 11:46:13 DEBUG --- http response --- 11:46:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:46:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:46:13 INFO [http_cmd]: http status code OK 11:46:13 DEBUG --- status code --- 11:46:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:13 DEBUG --- http response --- 11:46:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:13 INFO [http_cmd]: http status code OK 11:46:13 DEBUG --- status code --- 11:46:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:13 DEBUG --- http response --- 11:46:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:13 INFO [http_cmd]: http status code OK 11:46:13 DEBUG --- status code --- 11:46:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:13 DEBUG --- http response --- 11:46:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904773" 11:46:13 INFO [http_cmd]: http status code OK 11:46:13 DEBUG --- status code --- 11:46:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:13 DEBUG --- http response --- 11:46:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- status code --- 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- http response --- 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904774" 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG { 11:46:14 DEBUG { 11:46:14 DEBUG { 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG "data": { 11:46:14 DEBUG "data": { 11:46:14 DEBUG "data": { 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "result": [ 11:46:14 DEBUG "result": [ 11:46:14 DEBUG "result": [ 11:46:14 DEBUG { 11:46:14 DEBUG { 11:46:14 DEBUG { 11:46:14 DEBUG "metric": { 11:46:14 DEBUG "metric": { 11:46:14 DEBUG "metric": { 11:46:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:46:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:46:14 DEBUG "instance": "10.142.0.11:9100" 11:46:14 DEBUG }, 11:46:14 DEBUG }, 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 DEBUG "value": [ 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG [...] 11:46:14 DEBUG [...] 11:46:14 DEBUG [...] 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG "pod": "ds-cts-0" 11:46:14 DEBUG "pod": "ds-cts-0" 11:46:14 DEBUG "instance": "10.142.0.51:9100" 11:46:14 DEBUG }, 11:46:14 DEBUG }, 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 DEBUG "value": [ 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG "0" 11:46:14 DEBUG "593.6446972716403" 11:46:14 DEBUG "0.0004333333333382446" 11:46:14 DEBUG ] 11:46:14 DEBUG ] 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 DEBUG ] 11:46:14 DEBUG ] 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG { 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG "data": { 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "result": [ 11:46:14 DEBUG { 11:46:14 DEBUG "metric": { 11:46:14 DEBUG "pod": "ds-idrepo-0" 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG [...] 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG "758149120" 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904774" 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG { 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG "data": { 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "result": [ 11:46:14 DEBUG { 11:46:14 DEBUG "metric": { 11:46:14 DEBUG "instance": "10.142.0.11:9100" 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG [...] 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG "instance": "10.142.0.51:9100" 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG "1553049.6333333333" 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904774" 11:46:14 DEBUG { 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904774" 11:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904774" 11:46:14 DEBUG --- http response --- 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:46:14 DEBUG "data": { 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG { 11:46:14 DEBUG "result": [ 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG { 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG "data": { 11:46:14 DEBUG "metric": { 11:46:14 DEBUG { 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "instance": "10.142.0.11:9100" 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG "result": [ 11:46:14 DEBUG }, 11:46:14 DEBUG "data": { 11:46:14 DEBUG { 11:46:14 DEBUG "value": [ 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "metric": { 11:46:14 DEBUG 11:46:14 DEBUG "result": [ 11:46:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:46:14 DEBUG 11:46:14 DEBUG { 11:46:14 DEBUG }, 11:46:14 DEBUG 11:46:14 DEBUG "metric": { 11:46:14 DEBUG "value": [ 11:46:14 DEBUG [...] 11:46:14 DEBUG "instance": "10.142.0.11:9100" 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG }, 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG [...] 11:46:14 DEBUG "instance": "10.142.0.51:9100" 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG }, 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG 11:46:14 DEBUG "0.00033333333333397756" 11:46:14 DEBUG [...] 11:46:14 DEBUG "pod": "ds-cts-0" 11:46:14 DEBUG ] 11:46:14 DEBUG 11:46:14 DEBUG }, 11:46:14 DEBUG } 11:46:14 DEBUG 11:46:14 DEBUG "value": [ 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG 11:46:14 DEBUG } 11:46:14 DEBUG "0.0027864777709342074" 11:46:14 DEBUG "instance": "10.142.0.51:9100" 11:46:14 DEBUG ] 11:46:14 DEBUG }, 11:46:14 DEBUG } 11:46:14 DEBUG "value": [ 11:46:14 DEBUG ] 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG } 11:46:14 DEBUG "25395.199999999997" 11:46:14 DEBUG } 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG { 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG "data": { 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "result": [ 11:46:14 DEBUG { 11:46:14 DEBUG "metric": { 11:46:14 DEBUG "instance": "10.142.0.11:9100" 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG [...] 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG "instance": "10.142.0.51:9100" 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG 1735904774, 11:46:14 INFO [http_cmd]: http status code OK 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG "1534676.8333333328" 11:46:14 DEBUG --- status code --- 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG ] 11:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG } 11:46:14 DEBUG --- http response --- 11:46:14 DEBUG ] 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:46:14 DEBUG } 11:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG } 11:46:14 DEBUG ----- output ----- 11:46:14 DEBUG { 11:46:14 DEBUG { 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG "status": "success", 11:46:14 DEBUG "data": { 11:46:14 DEBUG "data": { 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "resultType": "vector", 11:46:14 DEBUG "result": [ 11:46:14 DEBUG "result": [ 11:46:14 DEBUG { 11:46:14 DEBUG { 11:46:14 DEBUG "metric": { 11:46:14 DEBUG "metric": { 11:46:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:46:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:46:14 DEBUG }, 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG [...] 11:46:14 DEBUG [...] 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG 11:46:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:46:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:46:14 DEBUG }, 11:46:14 DEBUG }, 11:46:14 DEBUG "value": [ 11:46:14 DEBUG "value": [ 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG 1735904774, 11:46:14 DEBUG "14990979072" 11:46:14 DEBUG "0.0342018494254187" 11:46:14 DEBUG ] 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 DEBUG ] 11:46:14 DEBUG ] 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:14 DEBUG } 11:46:17 INFO 11:46:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:46:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:46:17 INFO [loop_until]: OK (rc = 0) 11:46:17 DEBUG --- stdout --- 11:46:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 170m 1% 3856Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 590m 1% 19057Mi 67% 11:46:17 DEBUG --- stderr --- 11:46:17 DEBUG 11:46:21 INFO 11:46:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:46:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:46:22 INFO [loop_until]: OK (rc = 0) 11:46:22 DEBUG --- stdout --- 11:46:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 174m 13479Mi rcs-645c86fdd6-j4rsk 63m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:46:22 DEBUG --- stderr --- 11:46:22 DEBUG 11:47:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904833" 11:47:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904833" 11:47:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904833" 11:47:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904833" 11:47:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904833" 11:47:13 INFO [http_cmd]: http status code OK 11:47:13 DEBUG --- status code --- 11:47:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:13 DEBUG --- http response --- 11:47:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:13 INFO [http_cmd]: http status code OK 11:47:13 DEBUG --- status code --- 11:47:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:13 DEBUG --- http response --- 11:47:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:13 INFO [http_cmd]: http status code OK 11:47:13 DEBUG --- status code --- 11:47:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:13 DEBUG --- http response --- 11:47:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:13 ERROR [http_cmd]: ERROR 11:47:13 INFO [http_cmd]: http status code OK 11:47:13 ERROR --- status code --- 11:47:13 DEBUG --- status code --- 11:47:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:47:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:13 ERROR --- http response --- 11:47:13 DEBUG --- http response --- 11:47:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:47:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:47:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904833" 11:47:13 INFO [http_cmd]: http status code OK 11:47:13 DEBUG --- status code --- 11:47:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:13 DEBUG --- http response --- 11:47:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 DEBUG --- status code --- 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG --- http response --- 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG { 11:47:14 DEBUG { 11:47:14 DEBUG { 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG "data": { 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG "data": { 11:47:14 DEBUG "data": { 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG { 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG "result": [ 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG "result": [ 11:47:14 DEBUG "result": [ 11:47:14 DEBUG { 11:47:14 DEBUG "data": { 11:47:14 DEBUG { 11:47:14 DEBUG { 11:47:14 DEBUG "metric": { 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG "metric": { 11:47:14 DEBUG "metric": { 11:47:14 DEBUG "instance": "10.142.0.11:9100" 11:47:14 DEBUG "result": [ 11:47:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:47:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:47:14 DEBUG }, 11:47:14 DEBUG { 11:47:14 DEBUG }, 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904834" 11:47:14 DEBUG }, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG "metric": { 11:47:14 DEBUG "value": [ 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 11:47:14 DEBUG "pod": "ds-idrepo-0" 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG }, 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG [...] 11:47:14 DEBUG 11:47:14 DEBUG [...] 11:47:14 DEBUG [...] 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG [...] 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG "instance": "10.142.0.51:9100" 11:47:14 DEBUG 11:47:14 DEBUG "pod": "ds-cts-0" 11:47:14 DEBUG "pod": "ds-cts-0" 11:47:14 DEBUG }, 11:47:14 DEBUG 11:47:14 DEBUG }, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG }, 11:47:14 DEBUG 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG "0" 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG }, 11:47:14 DEBUG "0.00046666666666700016" 11:47:14 DEBUG ] 11:47:14 DEBUG "555.8231841774943" 11:47:14 DEBUG "value": [ 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG ] 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG } 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG "758149120" 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG ] 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG { 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG "data": { 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG "result": [ 11:47:14 DEBUG { 11:47:14 DEBUG "metric": { 11:47:14 DEBUG "instance": "10.142.0.11:9100" 11:47:14 DEBUG }, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG [...] 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG "instance": "10.142.0.51:9100" 11:47:14 DEBUG }, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG "2376364.5333333323" 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904834" 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904834" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG { 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG "data": { 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG "result": [ 11:47:14 DEBUG { 11:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904834" 11:47:14 DEBUG "metric": { 11:47:14 DEBUG "instance": "10.142.0.11:9100" 11:47:14 DEBUG }, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG [...] 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG "instance": "10.142.0.51:9100" 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG }, 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG "value": [ 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:47:14 DEBUG "0" 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG ] 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG { 11:47:14 DEBUG } 11:47:14 DEBUG { 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG ] 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG "data": { 11:47:14 DEBUG } 11:47:14 DEBUG "data": { 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG } 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG "result": [ 11:47:14 DEBUG "result": [ 11:47:14 DEBUG { 11:47:14 DEBUG { 11:47:14 DEBUG "metric": { 11:47:14 DEBUG "metric": { 11:47:14 DEBUG "instance": "10.142.0.11:9100" 11:47:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:47:14 DEBUG }, 11:47:14 DEBUG }, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG [...] 11:47:14 DEBUG [...] 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG "instance": "10.142.0.51:9100" 11:47:14 DEBUG "pod": "ds-cts-0" 11:47:14 DEBUG }, 11:47:14 DEBUG }, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG "25941.333333333332" 11:47:14 DEBUG "0.0027712793025103204" 11:47:14 DEBUG ] 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 DEBUG ] 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG { 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG "data": { 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG "result": [ 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:47:14 DEBUG { 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG "metric": { 11:47:14 DEBUG { 11:47:14 DEBUG "instance": "10.142.0.11:9100" 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG }, 11:47:14 DEBUG "data": { 11:47:14 DEBUG "value": [ 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG 11:47:14 DEBUG "result": [ 11:47:14 DEBUG 11:47:14 DEBUG { 11:47:14 DEBUG 11:47:14 DEBUG "metric": { 11:47:14 DEBUG [...] 11:47:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:47:14 DEBUG 11:47:14 DEBUG }, 11:47:14 DEBUG 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG "instance": "10.142.0.51:9100" 11:47:14 DEBUG 11:47:14 DEBUG }, 11:47:14 DEBUG 11:47:14 DEBUG "value": [ 11:47:14 DEBUG [...] 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG 11:47:14 DEBUG "2357661.3000000007" 11:47:14 DEBUG 11:47:14 DEBUG ] 11:47:14 DEBUG 11:47:14 DEBUG } 11:47:14 DEBUG ] 11:47:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:47:14 DEBUG } 11:47:14 DEBUG }, 11:47:14 DEBUG } 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG "0.09492995776054014" 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:14 INFO [http_cmd]: http status code OK 11:47:14 DEBUG --- status code --- 11:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:47:14 DEBUG --- http response --- 11:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:47:14 DEBUG ----- output ----- 11:47:14 DEBUG { 11:47:14 DEBUG "status": "success", 11:47:14 DEBUG "data": { 11:47:14 DEBUG "resultType": "vector", 11:47:14 DEBUG "result": [ 11:47:14 DEBUG { 11:47:14 DEBUG "metric": { 11:47:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:47:14 DEBUG }, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG [...] 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG 11:47:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:47:14 DEBUG }, 11:47:14 DEBUG "value": [ 11:47:14 DEBUG 1735904834, 11:47:14 DEBUG "14991138816" 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG ] 11:47:14 DEBUG } 11:47:14 DEBUG } 11:47:17 INFO 11:47:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:47:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:47:17 INFO [loop_until]: OK (rc = 0) 11:47:17 DEBUG --- stdout --- 11:47:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 216m 1% 3859Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 169m 0% 19052Mi 66% 11:47:17 DEBUG --- stderr --- 11:47:17 DEBUG 11:47:22 INFO 11:47:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:47:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:47:22 INFO [loop_until]: OK (rc = 0) 11:47:22 DEBUG --- stdout --- 11:47:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 84m 13479Mi rcs-645c86fdd6-j4rsk 83m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:47:22 DEBUG --- stderr --- 11:47:22 DEBUG 11:48:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904893" 11:48:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904893" 11:48:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904893" 11:48:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904893" 11:48:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904893" 11:48:13 INFO [http_cmd]: http status code OK 11:48:13 DEBUG --- status code --- 11:48:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:13 DEBUG --- http response --- 11:48:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:13 INFO [http_cmd]: http status code OK 11:48:13 DEBUG --- status code --- 11:48:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:13 DEBUG --- http response --- 11:48:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:13 INFO [http_cmd]: http status code OK 11:48:13 DEBUG --- status code --- 11:48:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:13 DEBUG --- http response --- 11:48:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:13 ERROR [http_cmd]: ERROR 11:48:13 ERROR --- status code --- 11:48:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:48:13 ERROR --- http response --- 11:48:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:48:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:48:13 INFO [http_cmd]: http status code OK 11:48:13 DEBUG --- status code --- 11:48:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:13 DEBUG --- http response --- 11:48:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904893" 11:48:13 INFO [http_cmd]: http status code OK 11:48:13 DEBUG --- status code --- 11:48:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:13 DEBUG --- http response --- 11:48:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG --- status code --- 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904894" 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904894" 11:48:14 DEBUG --- http response --- 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { 11:48:14 DEBUG { 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:48:14 DEBUG { 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG "data": { 11:48:14 DEBUG "data": { 11:48:14 DEBUG { 11:48:14 DEBUG "data": { 11:48:14 DEBUG { 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904894" 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG "result": [ 11:48:14 DEBUG "result": [ 11:48:14 DEBUG "data": { 11:48:14 DEBUG "result": [ 11:48:14 DEBUG "data": { 11:48:14 DEBUG { 11:48:14 DEBUG { 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG "result": [ 11:48:14 DEBUG { 11:48:14 DEBUG "metric": { 11:48:14 DEBUG "metric": { 11:48:14 DEBUG { 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG "result": [ 11:48:14 DEBUG "pod": "ds-cts-0" 11:48:14 DEBUG "pod": "ds-idrepo-0" 11:48:14 DEBUG "metric": { 11:48:14 DEBUG "metric": { 11:48:14 DEBUG { 11:48:14 DEBUG }, 11:48:14 DEBUG }, 11:48:14 DEBUG "value": [ 11:48:14 DEBUG "pod": "ds-cts-0" 11:48:14 DEBUG }, 11:48:14 DEBUG "metric": { 11:48:14 DEBUG "instance": "10.142.0.11:9100" 11:48:14 DEBUG 11:48:14 DEBUG "value": [ 11:48:14 DEBUG "value": [ 11:48:14 DEBUG "instance": "10.142.0.11:9100" 11:48:14 DEBUG }, 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG }, 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904894" 11:48:14 DEBUG "value": [ 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904894" 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG "value": [ 11:48:14 DEBUG 11:48:14 DEBUG [...] 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG [...] 11:48:14 DEBUG [...] 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904894" 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG [...] 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904894" 11:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904894" 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG [...] 11:48:14 DEBUG 11:48:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG }, 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:48:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG "value": [ 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG }, 11:48:14 DEBUG }, 11:48:14 DEBUG 11:48:14 DEBUG "instance": "10.142.0.51:9100" 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG --- http response --- 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG "value": [ 11:48:14 DEBUG "value": [ 11:48:14 DEBUG "instance": "10.142.0.51:9100" 11:48:14 DEBUG }, 11:48:14 DEBUG "758149120" 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG }, 11:48:14 DEBUG "value": [ 11:48:14 DEBUG ] 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG "0" 11:48:14 DEBUG "347.8457793126875" 11:48:14 DEBUG "value": [ 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG } 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG { 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG ] 11:48:14 DEBUG ] 11:48:14 DEBUG } 11:48:14 DEBUG ] 11:48:14 DEBUG } 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG } 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG "0.0004999999999995453" 11:48:14 INFO [http_cmd]: http status code OK 11:48:14 DEBUG ] 11:48:14 DEBUG } 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG "data": { 11:48:14 DEBUG ] 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG "2381657.1999999993" 11:48:14 DEBUG ] 11:48:14 DEBUG --- status code --- 11:48:14 DEBUG } 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG } 11:48:14 DEBUG { 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG } 11:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:48:14 DEBUG } 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG "result": [ 11:48:14 DEBUG } 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG ] 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:48:14 DEBUG ] 11:48:14 DEBUG --- http response --- 11:48:14 DEBUG { 11:48:14 DEBUG { 11:48:14 DEBUG "data": { 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:48:14 DEBUG } 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG } 11:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG "metric": { 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG ] 11:48:14 DEBUG { 11:48:14 DEBUG } 11:48:14 DEBUG ----- output ----- 11:48:14 DEBUG "data": { 11:48:14 DEBUG "instance": "10.142.0.11:9100" 11:48:14 DEBUG "result": [ 11:48:14 DEBUG { 11:48:14 DEBUG } 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG { 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG }, 11:48:14 DEBUG { 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG } 11:48:14 DEBUG "data": { 11:48:14 DEBUG "status": "success", 11:48:14 DEBUG "result": [ 11:48:14 DEBUG "value": [ 11:48:14 DEBUG "metric": { 11:48:14 DEBUG "data": { 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG "data": { 11:48:14 DEBUG { 11:48:14 DEBUG 11:48:14 DEBUG "instance": "10.142.0.11:9100" 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG "result": [ 11:48:14 DEBUG "resultType": "vector", 11:48:14 DEBUG "metric": { 11:48:14 DEBUG 11:48:14 DEBUG }, 11:48:14 DEBUG "result": [ 11:48:14 DEBUG { 11:48:14 DEBUG "result": [ 11:48:14 DEBUG "pod": "ds-cts-0" 11:48:14 DEBUG 11:48:14 DEBUG "value": [ 11:48:14 DEBUG { 11:48:14 DEBUG "metric": { 11:48:14 DEBUG { 11:48:14 DEBUG }, 11:48:14 DEBUG [...] 11:48:14 DEBUG 11:48:14 DEBUG "metric": { 11:48:14 DEBUG "instance": "10.142.0.11:9100" 11:48:14 DEBUG "metric": { 11:48:14 DEBUG "value": [ 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:48:14 DEBUG }, 11:48:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG }, 11:48:14 DEBUG "value": [ 11:48:14 DEBUG }, 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG [...] 11:48:14 DEBUG "value": [ 11:48:14 DEBUG 11:48:14 DEBUG "value": [ 11:48:14 DEBUG 11:48:14 DEBUG "instance": "10.142.0.51:9100" 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG [...] 11:48:14 DEBUG }, 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG "value": [ 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG [...] 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG "instance": "10.142.0.51:9100" 11:48:14 DEBUG [...] 11:48:14 DEBUG 11:48:14 DEBUG [...] 11:48:14 DEBUG 11:48:14 DEBUG "0" 11:48:14 DEBUG }, 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG "pod": "ds-idrepo-0" 11:48:14 DEBUG ] 11:48:14 DEBUG "value": [ 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG 11:48:14 DEBUG }, 11:48:14 DEBUG } 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG 11:48:14 DEBUG "instance": "10.142.0.51:9100" 11:48:14 DEBUG 11:48:14 DEBUG "value": [ 11:48:14 DEBUG ] 11:48:14 DEBUG "19524.266666666666" 11:48:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:48:14 DEBUG }, 11:48:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG } 11:48:14 DEBUG ] 11:48:14 DEBUG }, 11:48:14 DEBUG "value": [ 11:48:14 DEBUG }, 11:48:14 DEBUG "0" 11:48:14 DEBUG } 11:48:14 DEBUG } 11:48:14 DEBUG "value": [ 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG "value": [ 11:48:14 DEBUG ] 11:48:14 DEBUG ] 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG "2365007.7" 11:48:14 DEBUG 1735904894, 11:48:14 DEBUG } 11:48:14 DEBUG } 11:48:14 DEBUG "14989680640" 11:48:14 DEBUG ] 11:48:14 DEBUG "0.10652311554529638" 11:48:14 DEBUG ] 11:48:14 DEBUG } 11:48:14 DEBUG ] 11:48:14 DEBUG } 11:48:14 DEBUG ] 11:48:14 DEBUG } 11:48:14 DEBUG } 11:48:14 DEBUG ] 11:48:14 DEBUG } 11:48:14 DEBUG } 11:48:14 DEBUG ] 11:48:14 DEBUG } 11:48:14 DEBUG ] 11:48:14 DEBUG } 11:48:14 DEBUG } 11:48:14 DEBUG } 11:48:14 DEBUG } 11:48:14 DEBUG } 11:48:17 INFO 11:48:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:48:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:48:17 INFO [loop_until]: OK (rc = 0) 11:48:17 DEBUG --- stdout --- 11:48:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 223m 1% 3826Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 196m 0% 19058Mi 67% 11:48:17 DEBUG --- stderr --- 11:48:17 DEBUG 11:48:22 INFO 11:48:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:48:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:48:22 INFO [loop_until]: OK (rc = 0) 11:48:22 DEBUG --- stdout --- 11:48:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 118m 13478Mi rcs-645c86fdd6-j4rsk 141m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:48:22 DEBUG --- stderr --- 11:48:22 DEBUG 11:49:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735904953" 11:49:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904953" 11:49:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904953" 11:49:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904953" 11:49:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904953" 11:49:13 INFO [http_cmd]: http status code OK 11:49:13 DEBUG --- status code --- 11:49:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:13 DEBUG --- http response --- 11:49:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:13 ERROR [http_cmd]: ERROR 11:49:13 INFO [http_cmd]: http status code OK 11:49:13 INFO [http_cmd]: http status code OK 11:49:13 ERROR --- status code --- 11:49:13 DEBUG --- status code --- 11:49:13 DEBUG --- status code --- 11:49:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:49:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:13 DEBUG --- http response --- 11:49:13 ERROR --- http response --- 11:49:13 DEBUG --- http response --- 11:49:13 INFO [http_cmd]: http status code OK 11:49:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:49:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:13 DEBUG --- status code --- 11:49:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:49:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:13 DEBUG --- http response --- 11:49:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904953" 11:49:13 INFO [http_cmd]: http status code OK 11:49:13 DEBUG --- status code --- 11:49:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:13 DEBUG --- http response --- 11:49:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG --- http response --- 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904954" 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG { 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG "data": { 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG ----- output ----- 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904954" 11:49:14 DEBUG "result": [ 11:49:14 DEBUG { 11:49:14 DEBUG { 11:49:14 DEBUG "status": "success", 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904954" 11:49:14 DEBUG "metric": { 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735904954" 11:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735904954" 11:49:14 DEBUG "data": { 11:49:14 DEBUG --- status code --- 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG }, 11:49:14 DEBUG "result": [ 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG "value": [ 11:49:14 DEBUG { 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG { 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG "data": { 11:49:14 DEBUG "metric": { 11:49:14 DEBUG --- http response --- 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG "pod": "ds-idrepo-0" 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:49:14 DEBUG 11:49:14 DEBUG "result": [ 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG }, 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG 11:49:14 DEBUG { 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG "value": [ 11:49:14 DEBUG { 11:49:14 DEBUG --- http response --- 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG { 11:49:14 DEBUG [...] 11:49:14 DEBUG "metric": { 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:49:14 DEBUG --- status code --- 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG 11:49:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG 11:49:14 DEBUG "data": { 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG "data": { 11:49:14 DEBUG 11:49:14 DEBUG }, 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:49:14 DEBUG 11:49:14 DEBUG "resultType": "vector", 11:49:14 INFO [http_cmd]: http status code OK 11:49:14 DEBUG { 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG 11:49:14 DEBUG "value": [ 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG [...] 11:49:14 DEBUG "result": [ 11:49:14 DEBUG --- status code --- 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:49:14 DEBUG "result": [ 11:49:14 DEBUG "pod": "ds-cts-0" 11:49:14 DEBUG 11:49:14 DEBUG { 11:49:14 DEBUG 11:49:14 DEBUG { 11:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:49:14 DEBUG "data": { 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:49:14 DEBUG { 11:49:14 DEBUG }, 11:49:14 DEBUG 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG 11:49:14 DEBUG "metric": { 11:49:14 DEBUG --- http response --- 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG { 11:49:14 DEBUG { 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG "metric": { 11:49:14 DEBUG "value": [ 11:49:14 DEBUG 11:49:14 DEBUG "data": { 11:49:14 DEBUG 11:49:14 DEBUG "instance": "10.142.0.11:9100" 11:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:49:14 DEBUG "result": [ 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG { 11:49:14 DEBUG "instance": "10.142.0.11:9100" 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG [...] 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:49:14 DEBUG }, 11:49:14 DEBUG ----- output ----- 11:49:14 DEBUG { 11:49:14 DEBUG "data": { 11:49:14 DEBUG "data": { 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG }, 11:49:14 DEBUG "491.54804039021053" 11:49:14 DEBUG 11:49:14 DEBUG "result": [ 11:49:14 DEBUG }, 11:49:14 DEBUG "value": [ 11:49:14 DEBUG { 11:49:14 DEBUG "metric": { 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG "data": { 11:49:14 DEBUG "value": [ 11:49:14 DEBUG ] 11:49:14 DEBUG 11:49:14 DEBUG { 11:49:14 DEBUG "value": [ 11:49:14 DEBUG 11:49:14 DEBUG "status": "success", 11:49:14 DEBUG "instance": "10.142.0.11:9100" 11:49:14 DEBUG "result": [ 11:49:14 DEBUG "result": [ 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG 11:49:14 DEBUG } 11:49:14 DEBUG 11:49:14 DEBUG "metric": { 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG 11:49:14 DEBUG "data": { 11:49:14 DEBUG }, 11:49:14 DEBUG { 11:49:14 DEBUG { 11:49:14 DEBUG "result": [ 11:49:14 DEBUG { 11:49:14 DEBUG "metric": { 11:49:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:49:14 DEBUG }, 11:49:14 DEBUG "758149120" 11:49:14 DEBUG 11:49:14 DEBUG "resultType": "vector", 11:49:14 DEBUG "value": [ 11:49:14 DEBUG "metric": { 11:49:14 DEBUG "metric": { 11:49:14 DEBUG 11:49:14 DEBUG ] 11:49:14 DEBUG "pod": "ds-cts-0" 11:49:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:49:14 DEBUG "value": [ 11:49:14 DEBUG ] 11:49:14 DEBUG [...] 11:49:14 DEBUG "result": [ 11:49:14 DEBUG 11:49:14 DEBUG "instance": "10.142.0.11:9100" 11:49:14 DEBUG "instance": "10.142.0.11:9100" 11:49:14 DEBUG 11:49:14 DEBUG } 11:49:14 DEBUG }, 11:49:14 DEBUG }, 11:49:14 DEBUG 11:49:14 DEBUG } 11:49:14 DEBUG 11:49:14 DEBUG { 11:49:14 DEBUG 11:49:14 DEBUG }, 11:49:14 DEBUG }, 11:49:14 DEBUG [...] 11:49:14 DEBUG } 11:49:14 DEBUG "value": [ 11:49:14 DEBUG "value": [ 11:49:14 DEBUG 11:49:14 DEBUG ] 11:49:14 DEBUG 11:49:14 DEBUG "metric": { 11:49:14 DEBUG 11:49:14 DEBUG "value": [ 11:49:14 DEBUG "value": [ 11:49:14 DEBUG 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG } 11:49:14 DEBUG 11:49:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:49:14 DEBUG [...] 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG "0" 11:49:14 DEBUG 11:49:14 DEBUG [...] 11:49:14 DEBUG } 11:49:14 DEBUG "instance": "10.142.0.51:9100" 11:49:14 DEBUG }, 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG ] 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG }, 11:49:14 DEBUG "value": [ 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG "instance": "10.142.0.51:9100" 11:49:14 DEBUG } 11:49:14 DEBUG [...] 11:49:14 DEBUG 11:49:14 DEBUG "value": [ 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG [...] 11:49:14 DEBUG [...] 11:49:14 DEBUG }, 11:49:14 DEBUG ] 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG 11:49:14 DEBUG "instance": "10.142.0.51:9100" 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG "value": [ 11:49:14 DEBUG } 11:49:14 DEBUG 11:49:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:49:14 DEBUG }, 11:49:14 DEBUG "value": [ 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG 11:49:14 DEBUG 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG } 11:49:14 DEBUG 11:49:14 DEBUG "2372104.833333334" 11:49:14 DEBUG 11:49:14 DEBUG }, 11:49:14 DEBUG "14993244160" 11:49:14 DEBUG 11:49:14 DEBUG "instance": "10.142.0.51:9100" 11:49:14 DEBUG }, 11:49:14 DEBUG "value": [ 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG "24576" 11:49:14 DEBUG ] 11:49:14 DEBUG } 11:49:14 DEBUG ] 11:49:14 DEBUG } 11:49:14 DEBUG } 11:49:14 DEBUG ] 11:49:14 DEBUG [...] 11:49:14 DEBUG "value": [ 11:49:14 DEBUG ] 11:49:14 DEBUG 11:49:14 DEBUG "0.0004999999999995453" 11:49:14 DEBUG "pod": "ds-cts-0" 11:49:14 DEBUG } 11:49:14 DEBUG ] 11:49:14 DEBUG } 11:49:14 DEBUG } 11:49:14 DEBUG "instance": "10.142.0.51:9100" 11:49:14 DEBUG ] 11:49:14 DEBUG }, 11:49:14 DEBUG 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG } 11:49:14 DEBUG }, 11:49:14 DEBUG } 11:49:14 DEBUG "value": [ 11:49:14 DEBUG 11:49:14 DEBUG "0.0003333333333330302" 11:49:14 DEBUG ] 11:49:14 DEBUG "value": [ 11:49:14 DEBUG ] 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG 11:49:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:49:14 DEBUG ] 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG } 11:49:14 DEBUG "0.0018660764504661016" 11:49:14 DEBUG } 11:49:14 DEBUG } 11:49:14 DEBUG }, 11:49:14 DEBUG "2352963.033333334" 11:49:14 DEBUG } 11:49:14 DEBUG ] 11:49:14 DEBUG } 11:49:14 DEBUG "value": [ 11:49:14 DEBUG ] 11:49:14 DEBUG } 11:49:14 DEBUG ] 11:49:14 DEBUG } 11:49:14 DEBUG } 11:49:14 DEBUG ] 11:49:14 DEBUG 1735904954, 11:49:14 DEBUG } 11:49:14 DEBUG } 11:49:14 DEBUG "0.13350065758891944" 11:49:14 DEBUG ] 11:49:14 DEBUG } 11:49:14 DEBUG ] 11:49:14 DEBUG } 11:49:14 DEBUG } 11:49:14 DEBUG } 11:49:14 DEBUG ] 11:49:14 DEBUG } 11:49:14 DEBUG } 11:49:17 INFO 11:49:17 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:49:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:49:18 INFO [loop_until]: OK (rc = 0) 11:49:18 DEBUG --- stdout --- 11:49:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 227m 1% 3853Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 227m 0% 19059Mi 67% 11:49:18 DEBUG --- stderr --- 11:49:18 DEBUG 11:49:22 INFO 11:49:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:49:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:49:22 INFO [loop_until]: OK (rc = 0) 11:49:22 DEBUG --- stdout --- 11:49:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 108m 13479Mi rcs-645c86fdd6-j4rsk 123m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:49:22 DEBUG --- stderr --- 11:49:22 DEBUG 11:50:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905013" 11:50:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905013" 11:50:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905013" 11:50:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905013" 11:50:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905013" 11:50:13 INFO [http_cmd]: http status code OK 11:50:13 DEBUG --- status code --- 11:50:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:13 DEBUG --- http response --- 11:50:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:13 INFO [http_cmd]: http status code OK 11:50:13 DEBUG --- status code --- 11:50:13 INFO [http_cmd]: http status code OK 11:50:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:13 DEBUG --- http response --- 11:50:13 DEBUG --- status code --- 11:50:13 INFO [http_cmd]: http status code OK 11:50:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:13 DEBUG --- status code --- 11:50:13 DEBUG --- http response --- 11:50:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:13 DEBUG --- http response --- 11:50:13 ERROR [http_cmd]: ERROR 11:50:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:13 ERROR --- status code --- 11:50:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:50:13 ERROR --- http response --- 11:50:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:50:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:50:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905013" 11:50:13 INFO [http_cmd]: http status code OK 11:50:13 DEBUG --- status code --- 11:50:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:13 DEBUG --- http response --- 11:50:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG --- http response --- 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 DEBUG --- status code --- 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 DEBUG --- http response --- 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905014" 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG { 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905014" 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905014" 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG "data": { 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG "result": [ 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905014" 11:50:14 DEBUG --- status code --- 11:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905014" 11:50:14 DEBUG { 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG "metric": { 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:50:14 DEBUG }, 11:50:14 DEBUG --- http response --- 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG "value": [ 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG --- status code --- 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG ----- output ----- 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:50:14 DEBUG { 11:50:14 DEBUG 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG { 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG 11:50:14 DEBUG { 11:50:14 DEBUG --- http response --- 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG { 11:50:14 DEBUG "data": { 11:50:14 DEBUG 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG "data": { 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG [...] 11:50:14 DEBUG "data": { 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG "result": [ 11:50:14 DEBUG "data": { 11:50:14 DEBUG "result": [ 11:50:14 DEBUG 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG { 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG { 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:50:14 DEBUG { 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG { 11:50:14 DEBUG 11:50:14 DEBUG "result": [ 11:50:14 DEBUG "status": "success", 11:50:14 INFO [http_cmd]: http status code OK 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG { 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG "metric": { 11:50:14 DEBUG "result": [ 11:50:14 DEBUG "metric": { 11:50:14 DEBUG "pod": "ds-idrepo-0" 11:50:14 DEBUG { 11:50:14 DEBUG "data": { 11:50:14 DEBUG --- status code --- 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG "data": { 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG { 11:50:14 DEBUG "instance": "10.142.0.11:9100" 11:50:14 DEBUG { 11:50:14 DEBUG 11:50:14 DEBUG }, 11:50:14 DEBUG "metric": { 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:50:14 DEBUG { 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG "data": { 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG }, 11:50:14 DEBUG "metric": { 11:50:14 DEBUG "instance": "10.142.0.11:9100" 11:50:14 DEBUG "value": [ 11:50:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:50:14 DEBUG "result": [ 11:50:14 DEBUG --- http response --- 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG "result": [ 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG "data": { 11:50:14 DEBUG "value": [ 11:50:14 DEBUG "pod": "ds-idrepo-0" 11:50:14 DEBUG }, 11:50:14 DEBUG 11:50:14 DEBUG }, 11:50:14 DEBUG { 11:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:50:14 DEBUG "data": { 11:50:14 DEBUG { 11:50:14 DEBUG "result": [ 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG 11:50:14 DEBUG }, 11:50:14 DEBUG "value": [ 11:50:14 DEBUG 11:50:14 DEBUG "value": [ 11:50:14 DEBUG "metric": { 11:50:14 DEBUG ----- output ----- 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG "metric": { 11:50:14 DEBUG { 11:50:14 DEBUG "result": [ 11:50:14 DEBUG 11:50:14 DEBUG "value": [ 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG "instance": "10.142.0.11:9100" 11:50:14 DEBUG { 11:50:14 DEBUG "result": [ 11:50:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:50:14 DEBUG "metric": { 11:50:14 DEBUG { 11:50:14 DEBUG 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG 11:50:14 DEBUG [...] 11:50:14 DEBUG 11:50:14 DEBUG }, 11:50:14 DEBUG "status": "success", 11:50:14 DEBUG { 11:50:14 DEBUG }, 11:50:14 DEBUG "value": [ 11:50:14 DEBUG 11:50:14 DEBUG [...] 11:50:14 DEBUG "1460.4862236628849" 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG "value": [ 11:50:14 DEBUG "data": { 11:50:14 DEBUG "metric": { 11:50:14 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:50:14 DEBUG "metric": { 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG ] 11:50:14 DEBUG [...] 11:50:14 DEBUG 11:50:14 DEBUG [...] 11:50:14 DEBUG 11:50:14 DEBUG "resultType": "vector", 11:50:14 DEBUG "instance": "10.142.0.11:9100" 11:50:14 DEBUG }, 11:50:14 DEBUG }, 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG } 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG "result": [ 11:50:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:50:14 DEBUG "value": [ 11:50:14 DEBUG "value": [ 11:50:14 DEBUG [...] 11:50:14 DEBUG 11:50:14 DEBUG ] 11:50:14 DEBUG 11:50:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG { 11:50:14 DEBUG }, 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG "instance": "10.142.0.51:9100" 11:50:14 DEBUG } 11:50:14 DEBUG 11:50:14 DEBUG }, 11:50:14 DEBUG 11:50:14 DEBUG "pod": "ds-idrepo-0" 11:50:14 DEBUG "metric": { 11:50:14 DEBUG "value": [ 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG }, 11:50:14 DEBUG } 11:50:14 DEBUG "instance": "10.142.0.51:9100" 11:50:14 DEBUG "value": [ 11:50:14 DEBUG [...] 11:50:14 DEBUG }, 11:50:14 DEBUG "instance": "10.142.0.11:9100" 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG "value": [ 11:50:14 DEBUG }, 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG 11:50:14 DEBUG "value": [ 11:50:14 DEBUG }, 11:50:14 DEBUG 11:50:14 DEBUG [...] 11:50:14 DEBUG [...] 11:50:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG "value": [ 11:50:14 DEBUG "758149120" 11:50:14 DEBUG 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG "value": [ 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG }, 11:50:14 DEBUG "12424.533333333335" 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG ] 11:50:14 DEBUG 11:50:14 DEBUG "0" 11:50:14 DEBUG 11:50:14 DEBUG [...] 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG "value": [ 11:50:14 DEBUG ] 11:50:14 DEBUG "1411319.7000000002" 11:50:14 DEBUG } 11:50:14 DEBUG "instance": "10.142.0.51:9100" 11:50:14 DEBUG ] 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG } 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG 11:50:14 DEBUG "instance": "10.142.0.51:9100" 11:50:14 DEBUG "pod": "ds-idrepo-0" 11:50:14 DEBUG "14995783680" 11:50:14 DEBUG ] 11:50:14 DEBUG ] 11:50:14 DEBUG }, 11:50:14 DEBUG } 11:50:14 DEBUG 11:50:14 DEBUG 11:50:14 DEBUG }, 11:50:14 DEBUG }, 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG "value": [ 11:50:14 DEBUG ] 11:50:14 DEBUG [...] 11:50:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:50:14 DEBUG "value": [ 11:50:14 DEBUG "value": [ 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG } 11:50:14 DEBUG 11:50:14 DEBUG }, 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG ] 11:50:14 DEBUG "0.0004999999999995453" 11:50:14 DEBUG } 11:50:14 DEBUG 11:50:14 DEBUG "value": [ 11:50:14 DEBUG "0.0003333333333330302" 11:50:14 DEBUG "0.005772836304290863" 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG ] 11:50:14 DEBUG "instance": "10.142.0.51:9100" 11:50:14 DEBUG }, 11:50:14 DEBUG "value": [ 11:50:14 DEBUG 1735905014, 11:50:14 DEBUG "1392426.9" 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG "0.17017178963336643" 11:50:14 DEBUG ] 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG ] 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:14 DEBUG } 11:50:18 INFO 11:50:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:50:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:50:18 INFO [loop_until]: OK (rc = 0) 11:50:18 DEBUG --- stdout --- 11:50:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 203m 1% 3854Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 167m 0% 19060Mi 67% 11:50:18 DEBUG --- stderr --- 11:50:18 DEBUG 11:50:22 INFO 11:50:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:50:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:50:22 INFO [loop_until]: OK (rc = 0) 11:50:22 DEBUG --- stdout --- 11:50:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 87m 13478Mi rcs-645c86fdd6-j4rsk 99m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:50:22 DEBUG --- stderr --- 11:50:22 DEBUG 11:51:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905073" 11:51:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905073" 11:51:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905073" 11:51:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905073" 11:51:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905073" 11:51:13 INFO [http_cmd]: http status code OK 11:51:13 DEBUG --- status code --- 11:51:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:13 DEBUG --- http response --- 11:51:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:13 INFO [http_cmd]: http status code OK 11:51:13 DEBUG --- status code --- 11:51:13 ERROR [http_cmd]: ERROR 11:51:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:13 INFO [http_cmd]: http status code OK 11:51:13 ERROR --- status code --- 11:51:13 DEBUG --- http response --- 11:51:13 DEBUG --- status code --- 11:51:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:51:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:13 ERROR --- http response --- 11:51:13 DEBUG --- http response --- 11:51:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:51:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:51:13 INFO [http_cmd]: http status code OK 11:51:13 DEBUG --- status code --- 11:51:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:13 DEBUG --- http response --- 11:51:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905073" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 DEBUG --- status code --- 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 DEBUG --- http response --- 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- http response --- 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905074" 11:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905074" 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG { 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG "data": { 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG "result": [ 11:51:14 DEBUG { 11:51:14 DEBUG "metric": { 11:51:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:51:14 DEBUG }, 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG [...] 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG "pod": "ds-cts-0" 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG }, 11:51:14 DEBUG { 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG "value": [ 11:51:14 DEBUG "status": "success", 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- status code --- 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG "data": { 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG "327.10467651050874" 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG { 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG ] 11:51:14 DEBUG "result": [ 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG --- http response --- 11:51:14 INFO [http_cmd]: http status code OK 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG } 11:51:14 DEBUG { 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 DEBUG --- status code --- 11:51:14 DEBUG "data": { 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG ] 11:51:14 DEBUG "metric": { 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 DEBUG { 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG } 11:51:14 DEBUG "pod": "ds-idrepo-0" 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG { 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG { 11:51:14 DEBUG { 11:51:14 DEBUG --- http response --- 11:51:14 DEBUG "result": [ 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 DEBUG } 11:51:14 DEBUG }, 11:51:14 DEBUG "value": [ 11:51:14 DEBUG "data": { 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:51:14 DEBUG { 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG { 11:51:14 DEBUG 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG "data": { 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG "data": { 11:51:14 DEBUG "data": { 11:51:14 DEBUG ----- output ----- 11:51:14 DEBUG "metric": { 11:51:14 DEBUG { 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG 11:51:14 DEBUG "result": [ 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG { 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG { 11:51:14 DEBUG "instance": "10.142.0.11:9100" 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG "data": { 11:51:14 DEBUG 11:51:14 DEBUG { 11:51:14 DEBUG "result": [ 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG "result": [ 11:51:14 DEBUG "result": [ 11:51:14 DEBUG "status": "success", 11:51:14 DEBUG }, 11:51:14 DEBUG "data": { 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG [...] 11:51:14 DEBUG "metric": { 11:51:14 DEBUG { 11:51:14 DEBUG "data": { 11:51:14 DEBUG { 11:51:14 DEBUG { 11:51:14 DEBUG "data": { 11:51:14 DEBUG "value": [ 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG "result": [ 11:51:14 DEBUG 11:51:14 DEBUG "instance": "10.142.0.11:9100" 11:51:14 DEBUG "metric": { 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG "metric": { 11:51:14 DEBUG "metric": { 11:51:14 DEBUG "resultType": "vector", 11:51:14 DEBUG 11:51:14 DEBUG "result": [ 11:51:14 DEBUG { 11:51:14 DEBUG 11:51:14 DEBUG }, 11:51:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:51:14 DEBUG "result": [ 11:51:14 DEBUG "instance": "10.142.0.11:9100" 11:51:14 DEBUG "instance": "10.142.0.11:9100" 11:51:14 DEBUG "result": [ 11:51:14 DEBUG 11:51:14 DEBUG { 11:51:14 DEBUG "metric": { 11:51:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:51:14 DEBUG }, 11:51:14 DEBUG }, 11:51:14 DEBUG { 11:51:14 DEBUG }, 11:51:14 DEBUG }, 11:51:14 DEBUG { 11:51:14 DEBUG 11:51:14 DEBUG "metric": { 11:51:14 DEBUG "instance": "10.142.0.11:9100" 11:51:14 DEBUG "value": [ 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG "value": [ 11:51:14 DEBUG "value": [ 11:51:14 DEBUG "metric": { 11:51:14 DEBUG [...] 11:51:14 DEBUG 11:51:14 DEBUG }, 11:51:14 DEBUG 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG [...] 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 11:51:14 DEBUG "metric": { 11:51:14 DEBUG 11:51:14 DEBUG [...] 11:51:14 DEBUG 11:51:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:51:14 DEBUG 11:51:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG [...] 11:51:14 DEBUG 11:51:14 DEBUG }, 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG [...] 11:51:14 DEBUG }, 11:51:14 DEBUG [...] 11:51:14 DEBUG 11:51:14 DEBUG }, 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 11:51:14 DEBUG "instance": "10.142.0.51:9100" 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG "instance": "10.142.0.51:9100" 11:51:14 DEBUG [...] 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG }, 11:51:14 DEBUG "758149120" 11:51:14 DEBUG }, 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG 11:51:14 DEBUG }, 11:51:14 DEBUG 11:51:14 DEBUG "instance": "10.142.0.51:9100" 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG "20480" 11:51:14 DEBUG 11:51:14 DEBUG "instance": "10.142.0.51:9100" 11:51:14 DEBUG 11:51:14 DEBUG "pod": "ds-cts-0" 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 11:51:14 DEBUG }, 11:51:14 DEBUG ] 11:51:14 DEBUG "value": [ 11:51:14 DEBUG ] 11:51:14 DEBUG 11:51:14 DEBUG }, 11:51:14 DEBUG [...] 11:51:14 DEBUG }, 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG [...] 11:51:14 DEBUG "value": [ 11:51:14 DEBUG } 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG } 11:51:14 DEBUG "instance": "10.142.0.51:9100" 11:51:14 DEBUG "value": [ 11:51:14 DEBUG 11:51:14 DEBUG "value": [ 11:51:14 DEBUG "14992293888" 11:51:14 DEBUG 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG ] 11:51:14 DEBUG "1451584.9666666668" 11:51:14 DEBUG ] 11:51:14 DEBUG }, 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG ] 11:51:14 DEBUG 11:51:14 DEBUG "1432862.9" 11:51:14 DEBUG ] 11:51:14 DEBUG ] 11:51:14 DEBUG } 11:51:14 DEBUG "value": [ 11:51:14 DEBUG "0.00043333333333161285" 11:51:14 DEBUG 11:51:14 DEBUG "0" 11:51:14 DEBUG } 11:51:14 DEBUG 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG "0.000666666666666534" 11:51:14 DEBUG "pod": "ds-cts-0" 11:51:14 DEBUG ] 11:51:14 DEBUG ] 11:51:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:51:14 DEBUG } 11:51:14 DEBUG ] 11:51:14 DEBUG ] 11:51:14 DEBUG } 11:51:14 DEBUG ] 11:51:14 DEBUG }, 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG }, 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG "value": [ 11:51:14 DEBUG ] 11:51:14 DEBUG } 11:51:14 DEBUG "value": [ 11:51:14 DEBUG ] 11:51:14 DEBUG ] 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG } 11:51:14 DEBUG 1735905074, 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG "0.0017667858467320158" 11:51:14 DEBUG } 11:51:14 DEBUG "0.16064630601616828" 11:51:14 DEBUG ] 11:51:14 DEBUG } 11:51:14 DEBUG ] 11:51:14 DEBUG ] 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG ] 11:51:14 DEBUG ] 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:14 DEBUG } 11:51:18 INFO 11:51:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:51:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:51:18 INFO [loop_until]: OK (rc = 0) 11:51:18 DEBUG --- stdout --- 11:51:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 197m 1% 3857Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 468m 1% 19056Mi 67% 11:51:18 DEBUG --- stderr --- 11:51:18 DEBUG 11:51:22 INFO 11:51:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:51:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:51:22 INFO [loop_until]: OK (rc = 0) 11:51:22 DEBUG --- stdout --- 11:51:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 65m 13479Mi rcs-645c86fdd6-j4rsk 44m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:51:22 DEBUG --- stderr --- 11:51:22 DEBUG 11:52:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905133" 11:52:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905133" 11:52:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905133" 11:52:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905133" 11:52:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905133" 11:52:13 INFO [http_cmd]: http status code OK 11:52:13 DEBUG --- status code --- 11:52:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:13 DEBUG --- http response --- 11:52:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:13 INFO [http_cmd]: http status code OK 11:52:13 DEBUG --- status code --- 11:52:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:13 DEBUG --- http response --- 11:52:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:13 INFO [http_cmd]: http status code OK 11:52:13 DEBUG --- status code --- 11:52:13 INFO [http_cmd]: http status code OK 11:52:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:13 ERROR [http_cmd]: ERROR 11:52:13 DEBUG --- status code --- 11:52:13 DEBUG --- http response --- 11:52:13 ERROR --- status code --- 11:52:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:52:13 DEBUG --- http response --- 11:52:13 ERROR --- http response --- 11:52:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:52:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG { 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG "data": { 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG "result": [ 11:52:14 DEBUG { 11:52:14 DEBUG "metric": { 11:52:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:52:14 DEBUG }, 11:52:14 DEBUG "value": [ 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG [...] 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG "pod": "ds-cts-0" 11:52:14 DEBUG }, 11:52:14 DEBUG "value": [ 11:52:14 DEBUG 1735905134, 11:52:14 DEBUG "475.58780841799705" 11:52:14 DEBUG ] 11:52:14 DEBUG } 11:52:14 DEBUG ] 11:52:14 DEBUG } 11:52:14 DEBUG } 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905134" 11:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905134" 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG --- http response --- 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG { 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG { 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG { 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG "data": { 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG "result": [ 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG "data": { 11:52:14 DEBUG { 11:52:14 DEBUG "data": { 11:52:14 DEBUG "data": { 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:52:14 DEBUG "status": "success", 11:52:14 INFO [http_cmd]: http status code OK 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG "metric": { 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG { 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG "data": { 11:52:14 DEBUG --- status code --- 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG "result": [ 11:52:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:52:14 DEBUG "result": [ 11:52:14 DEBUG "result": [ 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG { 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG { 11:52:14 DEBUG }, 11:52:14 DEBUG { 11:52:14 DEBUG { 11:52:14 DEBUG "data": { 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG "result": [ 11:52:14 DEBUG --- http response --- 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:52:14 DEBUG "metric": { 11:52:14 DEBUG "value": [ 11:52:14 DEBUG "metric": { 11:52:14 DEBUG "metric": { 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG "data": { 11:52:14 DEBUG { 11:52:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG "instance": "10.142.0.11:9100" 11:52:14 DEBUG 11:52:14 DEBUG "instance": "10.142.0.11:9100" 11:52:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:52:14 DEBUG "result": [ 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG "metric": { 11:52:14 DEBUG ----- output ----- 11:52:14 DEBUG { 11:52:14 DEBUG { 11:52:14 DEBUG }, 11:52:14 DEBUG 11:52:14 DEBUG }, 11:52:14 DEBUG }, 11:52:14 DEBUG { 11:52:14 DEBUG "result": [ 11:52:14 DEBUG "pod": "ds-idrepo-0" 11:52:14 DEBUG { 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG "value": [ 11:52:14 DEBUG 11:52:14 DEBUG "value": [ 11:52:14 DEBUG "value": [ 11:52:14 DEBUG "metric": { 11:52:14 DEBUG { 11:52:14 DEBUG }, 11:52:14 DEBUG "status": "success", 11:52:14 DEBUG "data": { 11:52:14 DEBUG "data": { 11:52:14 DEBUG 11:52:14 DEBUG [...] 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:52:14 DEBUG "metric": { 11:52:14 DEBUG "value": [ 11:52:14 DEBUG "data": { 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG }, 11:52:14 DEBUG "instance": "10.142.0.11:9100" 11:52:14 DEBUG 11:52:14 DEBUG "resultType": "vector", 11:52:14 DEBUG "result": [ 11:52:14 DEBUG "result": [ 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG "value": [ 11:52:14 DEBUG }, 11:52:14 DEBUG 11:52:14 DEBUG "result": [ 11:52:14 DEBUG { 11:52:14 DEBUG { 11:52:14 DEBUG [...] 11:52:14 DEBUG 11:52:14 DEBUG [...] 11:52:14 DEBUG [...] 11:52:14 DEBUG 11:52:14 DEBUG "value": [ 11:52:14 DEBUG 11:52:14 DEBUG { 11:52:14 DEBUG "metric": { 11:52:14 DEBUG "metric": { 11:52:14 DEBUG 11:52:14 DEBUG "pod": "ds-cts-0" 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG [...] 11:52:14 DEBUG "metric": { 11:52:14 DEBUG "instance": "10.142.0.11:9100" 11:52:14 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:52:14 DEBUG 11:52:14 DEBUG }, 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG "instance": "10.142.0.11:9100" 11:52:14 DEBUG }, 11:52:14 DEBUG }, 11:52:14 DEBUG 11:52:14 DEBUG "value": [ 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG [...] 11:52:14 DEBUG 11:52:14 DEBUG [...] 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG "instance": "10.142.0.51:9100" 11:52:14 DEBUG 1735905134, 11:52:14 DEBUG "instance": "10.142.0.51:9100" 11:52:14 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:52:14 DEBUG 11:52:14 DEBUG 11:52:14 DEBUG }, 11:52:14 DEBUG "value": [ 11:52:14 DEBUG "value": [ 11:52:14 DEBUG "instance": "10.142.0.51:9100" 11:52:14 DEBUG }, 11:52:14 DEBUG "0" 11:52:14 DEBUG }, 11:52:14 DEBUG }, 11:52:14 DEBUG 11:52:14 DEBUG 11:52:15 DEBUG "value": [ 11:52:15 DEBUG 11:52:15 DEBUG 11:52:15 DEBUG }, 11:52:15 DEBUG "value": [ 11:52:15 DEBUG ] 11:52:15 DEBUG "value": [ 11:52:15 DEBUG "value": [ 11:52:15 DEBUG 11:52:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:52:15 DEBUG 11:52:15 DEBUG 11:52:15 DEBUG 11:52:15 DEBUG "value": [ 11:52:15 DEBUG 1735905134, 11:52:15 DEBUG } 11:52:15 DEBUG 1735905134, 11:52:15 DEBUG 1735905134, 11:52:15 DEBUG "pod": "ds-cts-0" 11:52:15 DEBUG }, 11:52:15 DEBUG "value": [ 11:52:15 DEBUG 1735905134, 11:52:15 DEBUG 11:52:15 DEBUG 1735905134, 11:52:15 DEBUG "31812.266666666666" 11:52:15 DEBUG ] 11:52:15 DEBUG "793795.7000000003" 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:15 DEBUG [...] 11:52:15 DEBUG "774645.9000000001" 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG "14991417344" 11:52:15 DEBUG }, 11:52:15 DEBUG 11:52:15 DEBUG 11:52:15 DEBUG "0.0027702467343896898" 11:52:15 DEBUG 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:15 DEBUG ] 11:52:15 DEBUG "value": [ 11:52:15 DEBUG 11:52:15 DEBUG [...] 11:52:15 DEBUG ] 11:52:15 DEBUG 11:52:15 DEBUG } 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG 1735905134, 11:52:15 DEBUG [...] 11:52:15 DEBUG 11:52:15 DEBUG } 11:52:15 DEBUG 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG ] 11:52:15 DEBUG "758149120" 11:52:15 DEBUG 11:52:15 DEBUG 11:52:15 DEBUG ] 11:52:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:15 DEBUG 11:52:15 DEBUG 11:52:15 DEBUG } 11:52:15 DEBUG }, 11:52:15 DEBUG } 11:52:15 DEBUG ] 11:52:15 DEBUG 11:52:15 DEBUG "instance": "10.142.0.51:9100" 11:52:15 DEBUG } 11:52:15 DEBUG "value": [ 11:52:15 DEBUG } 11:52:15 DEBUG "instance": "10.142.0.51:9100" 11:52:15 DEBUG }, 11:52:15 DEBUG 1735905134, 11:52:15 DEBUG ] 11:52:15 DEBUG }, 11:52:15 DEBUG "value": [ 11:52:15 DEBUG "0.6707678863070496" 11:52:15 DEBUG } 11:52:15 DEBUG "value": [ 11:52:15 DEBUG 1735905134, 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG 1735905134, 11:52:15 DEBUG "0.0010666666666679701" 11:52:15 DEBUG } 11:52:15 DEBUG "0.001000000000000038" 11:52:15 DEBUG ] 11:52:15 DEBUG ] 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG ] 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:15 DEBUG } 11:52:18 INFO 11:52:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:52:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:52:18 INFO [loop_until]: OK (rc = 0) 11:52:18 DEBUG --- stdout --- 11:52:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 171m 1% 3850Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 131m 0% 19054Mi 66% 11:52:18 DEBUG --- stderr --- 11:52:18 DEBUG 11:52:22 INFO 11:52:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:52:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:52:22 INFO [loop_until]: OK (rc = 0) 11:52:22 DEBUG --- stdout --- 11:52:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:52:22 DEBUG --- stderr --- 11:52:22 DEBUG 11:53:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905193" 11:53:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905193" 11:53:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905193" 11:53:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905193" 11:53:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905193" 11:53:13 INFO [http_cmd]: http status code OK 11:53:13 INFO [http_cmd]: http status code OK 11:53:13 INFO [http_cmd]: http status code OK 11:53:13 INFO [http_cmd]: http status code OK 11:53:13 DEBUG --- status code --- 11:53:13 ERROR [http_cmd]: ERROR 11:53:13 DEBUG --- status code --- 11:53:13 DEBUG --- status code --- 11:53:13 DEBUG --- status code --- 11:53:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:13 ERROR --- status code --- 11:53:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:13 DEBUG --- http response --- 11:53:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:53:13 DEBUG --- http response --- 11:53:13 DEBUG --- http response --- 11:53:13 DEBUG --- http response --- 11:53:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:13 ERROR --- http response --- 11:53:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:53:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905194" 11:53:14 INFO [http_cmd]: http status code OK 11:53:14 DEBUG --- status code --- 11:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:14 DEBUG --- http response --- 11:53:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:53:14 DEBUG ----- output ----- 11:53:14 DEBUG { 11:53:14 DEBUG "status": "success", 11:53:14 DEBUG "data": { 11:53:14 DEBUG "resultType": "vector", 11:53:14 DEBUG "result": [ 11:53:14 DEBUG { 11:53:14 DEBUG "metric": { 11:53:14 DEBUG "pod": "ds-cts-0" 11:53:14 DEBUG }, 11:53:14 DEBUG "value": [ 11:53:14 DEBUG 11:53:14 DEBUG 11:53:14 DEBUG 11:53:14 DEBUG [...] 11:53:14 DEBUG 11:53:14 DEBUG 11:53:14 DEBUG 11:53:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:53:14 DEBUG }, 11:53:14 DEBUG "value": [ 11:53:14 DEBUG 1735905194, 11:53:14 DEBUG "274.549232522287" 11:53:14 DEBUG ] 11:53:14 DEBUG } 11:53:14 DEBUG ] 11:53:14 DEBUG } 11:53:14 DEBUG } 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905195" 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905195" 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905195" 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905195" 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905195" 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905195" 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905195" 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905195" 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905195" 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 DEBUG --- status code --- 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG { 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG "data": { 11:53:15 DEBUG "resultType": "vector", 11:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905195" 11:53:15 DEBUG "result": [ 11:53:15 DEBUG { 11:53:15 DEBUG "metric": { 11:53:15 DEBUG "pod": "ds-cts-0" 11:53:15 DEBUG }, 11:53:15 DEBUG "value": [ 11:53:15 DEBUG 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 DEBUG 11:53:15 DEBUG --- status code --- 11:53:15 DEBUG 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 DEBUG [...] 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 DEBUG --- status code --- 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG --- status code --- 11:53:15 DEBUG --- status code --- 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 DEBUG 11:53:15 DEBUG --- status code --- 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 DEBUG --- status code --- 11:53:15 DEBUG 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 INFO [http_cmd]: http status code OK 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 DEBUG --- status code --- 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 DEBUG 11:53:15 DEBUG --- status code --- 11:53:15 DEBUG --- status code --- 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:53:15 DEBUG { 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG }, 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG --- http response --- 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:53:15 DEBUG { 11:53:15 DEBUG "data": { 11:53:15 DEBUG { 11:53:15 DEBUG { 11:53:15 DEBUG "value": [ 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:53:15 DEBUG { 11:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG "resultType": "vector", 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG { 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG "data": { 11:53:15 DEBUG { 11:53:15 DEBUG "data": { 11:53:15 DEBUG "result": [ 11:53:15 DEBUG "data": { 11:53:15 DEBUG "data": { 11:53:15 DEBUG "0" 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG { 11:53:15 DEBUG ----- output ----- 11:53:15 DEBUG "resultType": "vector", 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG "resultType": "vector", 11:53:15 DEBUG { 11:53:15 DEBUG "resultType": "vector", 11:53:15 DEBUG "resultType": "vector", 11:53:15 DEBUG ] 11:53:15 DEBUG "data": { 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG { 11:53:15 DEBUG "result": [ 11:53:15 DEBUG "data": { 11:53:15 DEBUG "result": [ 11:53:15 DEBUG "metric": { 11:53:15 DEBUG "result": [ 11:53:15 DEBUG "result": [ 11:53:15 DEBUG } 11:53:15 DEBUG "resultType": "vector", 11:53:15 DEBUG "data": { 11:53:15 DEBUG "resultType": "vector", 11:53:15 DEBUG "result": [ 11:53:15 DEBUG "resultType": "vector", 11:53:15 DEBUG { 11:53:15 DEBUG "instance": "10.142.0.11:9100" 11:53:15 DEBUG { 11:53:15 DEBUG { 11:53:15 DEBUG ] 11:53:15 DEBUG "result": [ 11:53:15 DEBUG "status": "success", 11:53:15 DEBUG { 11:53:15 DEBUG { 11:53:15 DEBUG "result": [ 11:53:15 DEBUG "metric": { 11:53:15 DEBUG }, 11:53:15 DEBUG "metric": { 11:53:15 DEBUG "metric": { 11:53:15 DEBUG } 11:53:15 DEBUG { 11:53:15 DEBUG "data": { 11:53:15 DEBUG "metric": { 11:53:15 DEBUG "metric": { 11:53:15 DEBUG { 11:53:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:53:15 DEBUG "value": [ 11:53:15 DEBUG "pod": "ds-cts-0" 11:53:15 DEBUG "instance": "10.142.0.11:9100" 11:53:15 DEBUG } 11:53:15 DEBUG "metric": { 11:53:15 DEBUG "resultType": "vector", 11:53:15 DEBUG "instance": "10.142.0.11:9100" 11:53:15 DEBUG "instance": "10.142.0.11:9100" 11:53:15 DEBUG "metric": { 11:53:15 DEBUG }, 11:53:15 DEBUG 11:53:15 DEBUG }, 11:53:15 DEBUG }, 11:53:15 DEBUG "instance": "10.142.0.11:9100" 11:53:15 DEBUG "result": [ 11:53:15 DEBUG }, 11:53:15 DEBUG }, 11:53:15 DEBUG "pod": "ds-idrepo-0" 11:53:15 DEBUG "value": [ 11:53:15 DEBUG 11:53:15 DEBUG "value": [ 11:53:15 DEBUG "value": [ 11:53:15 DEBUG }, 11:53:15 DEBUG { 11:53:15 DEBUG "value": [ 11:53:15 DEBUG "value": [ 11:53:15 DEBUG }, 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG "value": [ 11:53:15 DEBUG "metric": { 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG "value": [ 11:53:15 DEBUG 11:53:15 DEBUG [...] 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG }, 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG [...] 11:53:15 DEBUG [...] 11:53:15 DEBUG 11:53:15 DEBUG [...] 11:53:15 DEBUG [...] 11:53:15 DEBUG 11:53:15 DEBUG "value": [ 11:53:15 DEBUG [...] 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG [...] 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG "instance": "10.142.0.51:9100" 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG [...] 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG }, 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG 11:53:15 DEBUG "instance": "10.142.0.51:9100" 11:53:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:53:15 DEBUG "value": [ 11:53:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:53:15 DEBUG "instance": "10.142.0.51:9100" 11:53:15 DEBUG 11:53:15 DEBUG [...] 11:53:15 DEBUG "instance": "10.142.0.51:9100" 11:53:15 DEBUG 11:53:15 DEBUG }, 11:53:15 DEBUG }, 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG }, 11:53:15 DEBUG }, 11:53:15 DEBUG "instance": "10.142.0.51:9100" 11:53:15 DEBUG 11:53:15 DEBUG }, 11:53:15 DEBUG 11:53:15 DEBUG "value": [ 11:53:15 DEBUG "value": [ 11:53:15 DEBUG "96244.26666666666" 11:53:15 DEBUG "value": [ 11:53:15 DEBUG "value": [ 11:53:15 DEBUG }, 11:53:15 DEBUG 11:53:15 DEBUG "value": [ 11:53:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG ] 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG "0.0005145787251156944" 11:53:15 DEBUG ] 11:53:15 DEBUG 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG }, 11:53:15 DEBUG "0.0005333333333320903" 11:53:15 DEBUG "14988787712" 11:53:15 DEBUG } 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG "value": [ 11:53:15 DEBUG } 11:53:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:53:15 DEBUG "79149.4" 11:53:15 DEBUG "value": [ 11:53:15 DEBUG ] 11:53:15 DEBUG ] 11:53:15 DEBUG ] 11:53:15 DEBUG "24029.866666666665" 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG ] 11:53:15 DEBUG }, 11:53:15 DEBUG ] 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG ] 11:53:15 DEBUG "0.00033333333333350387" 11:53:15 DEBUG } 11:53:15 DEBUG "value": [ 11:53:15 DEBUG } 11:53:15 DEBUG "758149120" 11:53:15 DEBUG ] 11:53:15 DEBUG ] 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG ] 11:53:15 DEBUG } 11:53:15 DEBUG 1735905195, 11:53:15 DEBUG ] 11:53:15 DEBUG ] 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG ] 11:53:15 DEBUG } 11:53:15 DEBUG "0.12354985245247575" 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG ] 11:53:15 DEBUG ] 11:53:15 DEBUG } 11:53:15 DEBUG ] 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:15 DEBUG ] 11:53:15 DEBUG } 11:53:15 DEBUG } 11:53:18 INFO 11:53:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:53:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:53:18 INFO [loop_until]: OK (rc = 0) 11:53:18 DEBUG --- stdout --- 11:53:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 110m 0% 3852Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 77m 0% 19056Mi 67% 11:53:18 DEBUG --- stderr --- 11:53:18 DEBUG 11:53:22 INFO 11:53:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:53:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:53:22 INFO [loop_until]: OK (rc = 0) 11:53:22 DEBUG --- stdout --- 11:53:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 2m 361Mi 11:53:22 DEBUG --- stderr --- 11:53:22 DEBUG 11:54:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905253" 11:54:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905253" 11:54:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905253" 11:54:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905253" 11:54:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905253" 11:54:13 INFO [http_cmd]: http status code OK 11:54:13 DEBUG --- status code --- 11:54:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:13 DEBUG --- http response --- 11:54:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:13 INFO [http_cmd]: http status code OK 11:54:13 INFO [http_cmd]: http status code OK 11:54:13 ERROR [http_cmd]: ERROR 11:54:13 INFO [http_cmd]: http status code OK 11:54:13 DEBUG --- status code --- 11:54:13 DEBUG --- status code --- 11:54:13 ERROR --- status code --- 11:54:13 DEBUG --- status code --- 11:54:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:54:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:13 DEBUG --- http response --- 11:54:13 DEBUG --- http response --- 11:54:13 ERROR --- http response --- 11:54:13 DEBUG --- http response --- 11:54:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:54:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905254" 11:54:14 INFO [http_cmd]: http status code OK 11:54:14 DEBUG --- status code --- 11:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:14 DEBUG --- http response --- 11:54:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:54:14 DEBUG ----- output ----- 11:54:14 DEBUG { 11:54:14 DEBUG "status": "success", 11:54:14 DEBUG "data": { 11:54:14 DEBUG "resultType": "vector", 11:54:14 DEBUG "result": [ 11:54:14 DEBUG { 11:54:14 DEBUG "metric": { 11:54:14 DEBUG "pod": "ds-cts-0" 11:54:14 DEBUG }, 11:54:14 DEBUG "value": [ 11:54:14 DEBUG 11:54:14 DEBUG 11:54:14 DEBUG 11:54:14 DEBUG [...] 11:54:14 DEBUG 11:54:14 DEBUG 11:54:14 DEBUG 11:54:14 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:54:14 DEBUG }, 11:54:14 DEBUG "value": [ 11:54:14 DEBUG 1735905254, 11:54:14 DEBUG "460.2634470142428" 11:54:14 DEBUG ] 11:54:14 DEBUG } 11:54:14 DEBUG ] 11:54:14 DEBUG } 11:54:14 DEBUG } 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905255" 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 DEBUG --- status code --- 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG --- http response --- 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG { 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG "data": { 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG "result": [ 11:54:15 DEBUG { 11:54:15 DEBUG "metric": { 11:54:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:54:15 DEBUG }, 11:54:15 DEBUG "value": [ 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905255" 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG [...] 11:54:15 DEBUG 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905255" 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:54:15 DEBUG }, 11:54:15 DEBUG "value": [ 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG "0" 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905255" 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905255" 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905255" 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905255" 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905255" 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905255" 11:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905255" 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 DEBUG --- status code --- 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG --- http response --- 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG { 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG "data": { 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG "result": [ 11:54:15 DEBUG { 11:54:15 DEBUG "metric": { 11:54:15 DEBUG "instance": "10.142.0.11:9100" 11:54:15 DEBUG }, 11:54:15 DEBUG "value": [ 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG [...] 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG "instance": "10.142.0.51:9100" 11:54:15 DEBUG }, 11:54:15 DEBUG "value": [ 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG "91682.86666666665" 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 DEBUG ] 11:54:15 DEBUG --- status code --- 11:54:15 DEBUG --- status code --- 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 DEBUG } 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG --- status code --- 11:54:15 DEBUG } 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 DEBUG --- http response --- 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 DEBUG --- http response --- 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG --- status code --- 11:54:15 DEBUG --- status code --- 11:54:15 DEBUG --- status code --- 11:54:15 DEBUG --- status code --- 11:54:15 INFO [http_cmd]: http status code OK 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:54:15 DEBUG --- http response --- 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG --- status code --- 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG --- http response --- 11:54:15 DEBUG --- http response --- 11:54:15 DEBUG --- http response --- 11:54:15 DEBUG --- http response --- 11:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:54:15 DEBUG { 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG { 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:54:15 DEBUG --- http response --- 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG { 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:54:15 DEBUG "data": { 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG "data": { 11:54:15 DEBUG { 11:54:15 DEBUG { 11:54:15 DEBUG { 11:54:15 DEBUG { 11:54:15 DEBUG ----- output ----- 11:54:15 DEBUG { 11:54:15 DEBUG "data": { 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG "status": "success", 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG "result": [ 11:54:15 DEBUG "data": { 11:54:15 DEBUG "data": { 11:54:15 DEBUG "data": { 11:54:15 DEBUG "data": { 11:54:15 DEBUG "result": [ 11:54:15 DEBUG "data": { 11:54:15 DEBUG "result": [ 11:54:15 DEBUG { 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG { 11:54:15 DEBUG "resultType": "vector", 11:54:15 DEBUG { 11:54:15 DEBUG "metric": { 11:54:15 DEBUG "result": [ 11:54:15 DEBUG "result": [ 11:54:15 DEBUG "result": [ 11:54:15 DEBUG "result": [ 11:54:15 DEBUG "metric": { 11:54:15 DEBUG "result": [ 11:54:15 DEBUG "metric": { 11:54:15 DEBUG "instance": "10.142.0.11:9100" 11:54:15 DEBUG { 11:54:15 DEBUG { 11:54:15 DEBUG { 11:54:15 DEBUG { 11:54:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:54:15 DEBUG { 11:54:15 DEBUG "instance": "10.142.0.11:9100" 11:54:15 DEBUG }, 11:54:15 DEBUG "metric": { 11:54:15 DEBUG "metric": { 11:54:15 DEBUG "metric": { 11:54:15 DEBUG "metric": { 11:54:15 DEBUG }, 11:54:15 DEBUG "metric": { 11:54:15 DEBUG }, 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:54:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:54:15 DEBUG "instance": "10.142.0.11:9100" 11:54:15 DEBUG "pod": "ds-idrepo-0" 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "instance": "10.142.0.11:9100" 11:54:15 DEBUG "value": [ 11:54:15 DEBUG 11:54:15 DEBUG }, 11:54:15 DEBUG }, 11:54:15 DEBUG }, 11:54:15 DEBUG }, 11:54:15 DEBUG 11:54:15 DEBUG }, 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "value": [ 11:54:15 DEBUG 11:54:15 DEBUG "value": [ 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG [...] 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG [...] 11:54:15 DEBUG 11:54:15 DEBUG [...] 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG [...] 11:54:15 DEBUG [...] 11:54:15 DEBUG [...] 11:54:15 DEBUG [...] 11:54:15 DEBUG 11:54:15 DEBUG [...] 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG "instance": "10.142.0.51:9100" 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:54:15 DEBUG 11:54:15 DEBUG "instance": "10.142.0.51:9100" 11:54:15 DEBUG }, 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG 11:54:15 DEBUG }, 11:54:15 DEBUG 11:54:15 DEBUG }, 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:54:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:54:15 DEBUG "instance": "10.142.0.51:9100" 11:54:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "instance": "10.142.0.51:9100" 11:54:15 DEBUG "value": [ 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG }, 11:54:15 DEBUG }, 11:54:15 DEBUG }, 11:54:15 DEBUG }, 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG }, 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG "0.00043333333333445503" 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "value": [ 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG "0.0008018473500588492" 11:54:15 DEBUG "72768.76666666666" 11:54:15 DEBUG ] 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG "value": [ 11:54:15 DEBUG "758550528" 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG "14990020608" 11:54:15 DEBUG "0" 11:54:15 DEBUG 1735905255, 11:54:15 DEBUG ] 11:54:15 DEBUG ] 11:54:15 DEBUG "0.11527577937326237" 11:54:15 DEBUG } 11:54:15 DEBUG ] 11:54:15 DEBUG ] 11:54:15 DEBUG ] 11:54:15 DEBUG "21299.199999999997" 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG ] 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG ] 11:54:15 DEBUG ] 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:15 DEBUG } 11:54:18 INFO 11:54:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:54:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:54:18 INFO [loop_until]: OK (rc = 0) 11:54:18 DEBUG --- stdout --- 11:54:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3872Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 78m 0% 19056Mi 67% 11:54:18 DEBUG --- stderr --- 11:54:18 DEBUG 11:54:22 INFO 11:54:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:54:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:54:22 INFO [loop_until]: OK (rc = 0) 11:54:22 DEBUG --- stdout --- 11:54:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:54:22 DEBUG --- stderr --- 11:54:22 DEBUG 11:55:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905313" 11:55:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905313" 11:55:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905313" 11:55:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905313" 11:55:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905313" 11:55:13 INFO [http_cmd]: http status code OK 11:55:13 DEBUG --- status code --- 11:55:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:13 DEBUG --- http response --- 11:55:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:13 INFO [http_cmd]: http status code OK 11:55:13 INFO [http_cmd]: http status code OK 11:55:13 ERROR [http_cmd]: ERROR 11:55:13 DEBUG --- status code --- 11:55:13 DEBUG --- status code --- 11:55:13 ERROR --- status code --- 11:55:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:13 INFO [http_cmd]: http status code OK 11:55:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:55:13 DEBUG --- http response --- 11:55:13 DEBUG --- http response --- 11:55:13 DEBUG --- status code --- 11:55:13 ERROR --- http response --- 11:55:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:55:13 DEBUG --- http response --- 11:55:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:55:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: http status code OK 11:55:14 DEBUG --- status code --- 11:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:14 DEBUG --- http response --- 11:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905314" 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG { 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG "data": { 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG "result": [ 11:55:15 DEBUG { 11:55:15 DEBUG "metric": { 11:55:15 DEBUG "pod": "ds-cts-0" 11:55:15 DEBUG }, 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG [...] 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:55:15 DEBUG }, 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 1735905314, 11:55:15 DEBUG "542.1296633901712" 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905315" 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905315" 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG { 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG "data": { 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG "result": [ 11:55:15 DEBUG { 11:55:15 DEBUG "metric": { 11:55:15 DEBUG "pod": "ds-cts-0" 11:55:15 DEBUG }, 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG [...] 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:55:15 DEBUG }, 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG "0" 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG { 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG "data": { 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG "result": [ 11:55:15 DEBUG { 11:55:15 DEBUG "metric": { 11:55:15 DEBUG "instance": "10.142.0.11:9100" 11:55:15 DEBUG }, 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG [...] 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "instance": "10.142.0.51:9100" 11:55:15 DEBUG }, 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG "89515.13333333332" 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905315" 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905315" 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905315" 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905315" 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905315" 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905315" 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905315" 11:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905315" 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 DEBUG --- status code --- 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:55:15 DEBUG --- http response --- 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 DEBUG ----- output ----- 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG { 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG { 11:55:15 DEBUG { 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG "data": { 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:55:15 DEBUG "data": { 11:55:15 DEBUG "data": { 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG "result": [ 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG "resultType": "vector", 11:55:15 INFO [http_cmd]: http status code OK 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG { 11:55:15 DEBUG { 11:55:15 DEBUG { 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG { 11:55:15 DEBUG "result": [ 11:55:15 DEBUG --- status code --- 11:55:15 DEBUG "result": [ 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG "metric": { 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG { 11:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:55:15 DEBUG { 11:55:15 DEBUG "data": { 11:55:15 DEBUG "data": { 11:55:15 DEBUG "instance": "10.142.0.11:9100" 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG "data": { 11:55:15 DEBUG "metric": { 11:55:15 DEBUG --- http response --- 11:55:15 DEBUG "metric": { 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG }, 11:55:15 DEBUG { 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG "instance": "10.142.0.11:9100" 11:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:55:15 DEBUG "pod": "ds-idrepo-0" 11:55:15 DEBUG "result": [ 11:55:15 DEBUG "result": [ 11:55:15 DEBUG "value": [ 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG "result": [ 11:55:15 DEBUG }, 11:55:15 DEBUG ----- output ----- 11:55:15 DEBUG }, 11:55:15 DEBUG { 11:55:15 DEBUG { 11:55:15 DEBUG 11:55:15 DEBUG "data": { 11:55:15 DEBUG { 11:55:15 DEBUG "value": [ 11:55:15 DEBUG { 11:55:15 DEBUG "value": [ 11:55:15 DEBUG "metric": { 11:55:15 DEBUG "metric": { 11:55:15 DEBUG 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG "metric": { 11:55:15 DEBUG 11:55:15 DEBUG "status": "success", 11:55:15 DEBUG 11:55:15 DEBUG "instance": "10.142.0.11:9100" 11:55:15 DEBUG "instance": "10.142.0.11:9100" 11:55:15 DEBUG 11:55:15 DEBUG "result": [ 11:55:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:55:15 DEBUG 11:55:15 DEBUG "data": { 11:55:15 DEBUG 11:55:15 DEBUG }, 11:55:15 DEBUG }, 11:55:15 DEBUG [...] 11:55:15 DEBUG { 11:55:15 DEBUG }, 11:55:15 DEBUG 11:55:15 DEBUG "resultType": "vector", 11:55:15 DEBUG 11:55:15 DEBUG "value": [ 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 11:55:15 DEBUG "metric": { 11:55:15 DEBUG "value": [ 11:55:15 DEBUG [...] 11:55:15 DEBUG "result": [ 11:55:15 DEBUG [...] 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "pod": "ds-cts-0" 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG { 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG }, 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "metric": { 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "instance": "10.142.0.51:9100" 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:55:15 DEBUG 11:55:15 DEBUG [...] 11:55:15 DEBUG [...] 11:55:15 DEBUG }, 11:55:15 DEBUG 11:55:15 DEBUG [...] 11:55:15 DEBUG "instance": "10.142.0.51:9100" 11:55:15 DEBUG }, 11:55:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG }, 11:55:15 DEBUG "value": [ 11:55:15 DEBUG }, 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG 11:55:15 DEBUG [...] 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG "0.0004666666666745793" 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG "24576" 11:55:15 DEBUG 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG "instance": "10.142.0.51:9100" 11:55:15 DEBUG "instance": "10.142.0.51:9100" 11:55:15 DEBUG ] 11:55:15 DEBUG 11:55:15 DEBUG 11:55:15 DEBUG ] 11:55:15 DEBUG 11:55:15 DEBUG "758542336" 11:55:15 DEBUG }, 11:55:15 DEBUG }, 11:55:15 DEBUG } 11:55:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:55:15 DEBUG 11:55:15 DEBUG } 11:55:15 DEBUG 11:55:15 DEBUG ] 11:55:15 DEBUG "value": [ 11:55:15 DEBUG "value": [ 11:55:15 DEBUG ] 11:55:15 DEBUG }, 11:55:15 DEBUG [...] 11:55:15 DEBUG ] 11:55:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:55:15 DEBUG } 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG } 11:55:15 DEBUG "value": [ 11:55:15 DEBUG 11:55:15 DEBUG } 11:55:15 DEBUG }, 11:55:15 DEBUG "value": [ 11:55:15 DEBUG "0.00033333333333397756" 11:55:15 DEBUG "70835.10000000002" 11:55:15 DEBUG } 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG 11:55:15 DEBUG } 11:55:15 DEBUG ] 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG ] 11:55:15 DEBUG ] 11:55:15 DEBUG "0.036383366173444356" 11:55:15 DEBUG 11:55:15 DEBUG } 11:55:15 DEBUG "0.0012009216662591197" 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:15 DEBUG ] 11:55:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:55:15 DEBUG } 11:55:15 DEBUG ] 11:55:15 DEBUG ] 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG }, 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:15 DEBUG ] 11:55:15 DEBUG "value": [ 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:15 DEBUG 1735905315, 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:15 DEBUG "14989815808" 11:55:15 DEBUG } 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG ] 11:55:15 DEBUG } 11:55:15 DEBUG } 11:55:18 INFO 11:55:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:55:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:55:18 INFO [loop_until]: OK (rc = 0) 11:55:18 DEBUG --- stdout --- 11:55:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3873Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 77m 0% 19055Mi 67% 11:55:18 DEBUG --- stderr --- 11:55:18 DEBUG 11:55:22 INFO 11:55:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:55:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:55:22 INFO [loop_until]: OK (rc = 0) 11:55:22 DEBUG --- stdout --- 11:55:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 13m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:55:22 DEBUG --- stderr --- 11:55:22 DEBUG 11:56:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905373" 11:56:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905373" 11:56:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905373" 11:56:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905373" 11:56:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905373" 11:56:13 INFO [http_cmd]: http status code OK 11:56:13 DEBUG --- status code --- 11:56:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:13 DEBUG --- http response --- 11:56:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:13 ERROR [http_cmd]: ERROR 11:56:13 ERROR --- status code --- 11:56:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:56:13 ERROR --- http response --- 11:56:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:56:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:56:13 INFO [http_cmd]: http status code OK 11:56:13 INFO [http_cmd]: http status code OK 11:56:13 INFO [http_cmd]: http status code OK 11:56:13 DEBUG --- status code --- 11:56:13 DEBUG --- status code --- 11:56:13 DEBUG --- status code --- 11:56:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:13 DEBUG --- http response --- 11:56:13 DEBUG --- http response --- 11:56:13 DEBUG --- http response --- 11:56:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905374" 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG --- status code --- 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 DEBUG --- http response --- 11:56:14 INFO [http_cmd]: http status code OK 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:14 DEBUG --- status code --- 11:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:14 DEBUG --- http response --- 11:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905375" 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735905375, "750.7157811592109" ] } ] } } 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905375" 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905375" 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735905375, "0" ] } ] } } 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:56:15 DEBUG ----- output ----- 11:56:15 DEBUG { 11:56:15 DEBUG "status": "success", 11:56:15 DEBUG "data": { 11:56:15 DEBUG "resultType": "vector", 11:56:15 DEBUG "result": [ 11:56:15 DEBUG { 11:56:15 DEBUG "metric": { 11:56:15 DEBUG "instance": "10.142.0.11:9100" 11:56:15 DEBUG }, 11:56:15 DEBUG "value": [ 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG [...] 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG "instance": "10.142.0.51:9100" 11:56:15 DEBUG }, 11:56:15 DEBUG "value": [ 11:56:15 DEBUG 1735905375, 11:56:15 DEBUG "1033324.5999999997" 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905375" 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905375" 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905375" 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905375" 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905375" 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905375" 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905375" 11:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905375" 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:56:15 DEBUG ----- output ----- 11:56:15 DEBUG { 11:56:15 DEBUG "status": "success", 11:56:15 DEBUG "data": { 11:56:15 DEBUG "resultType": "vector", 11:56:15 DEBUG "result": [ 11:56:15 DEBUG { 11:56:15 DEBUG "metric": { 11:56:15 DEBUG "instance": "10.142.0.11:9100" 11:56:15 DEBUG }, 11:56:15 DEBUG "value": [ 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG [...] 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG "instance": "10.142.0.51:9100" 11:56:15 DEBUG }, 11:56:15 DEBUG "value": [ 11:56:15 DEBUG 1735905375, 11:56:15 DEBUG "0.0003333333333330302" 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:56:15 DEBUG ----- output ----- 11:56:15 DEBUG { 11:56:15 DEBUG "status": "success", 11:56:15 DEBUG "data": { 11:56:15 DEBUG "resultType": "vector", 11:56:15 DEBUG "result": [ 11:56:15 DEBUG { 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 DEBUG "metric": { 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG "instance": "10.142.0.11:9100" 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG }, 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG "value": [ 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG 11:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 INFO [http_cmd]: http status code OK 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG 11:56:15 DEBUG ----- output ----- 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG --- status code --- 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG 11:56:15 DEBUG { 11:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:56:15 DEBUG [...] 11:56:15 DEBUG "status": "success", 11:56:15 DEBUG ----- output ----- 11:56:15 DEBUG --- http response --- 11:56:15 DEBUG ----- output ----- 11:56:15 DEBUG ----- output ----- 11:56:15 DEBUG ----- output ----- 11:56:15 DEBUG 11:56:15 DEBUG "data": { 11:56:15 DEBUG { 11:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:56:15 DEBUG { 11:56:15 DEBUG { 11:56:15 DEBUG { 11:56:15 DEBUG 11:56:15 DEBUG "resultType": "vector", 11:56:15 DEBUG "status": "success", 11:56:15 DEBUG ----- output ----- 11:56:15 DEBUG "status": "success", 11:56:15 DEBUG "status": "success", 11:56:15 DEBUG "status": "success", 11:56:15 DEBUG 11:56:15 DEBUG "result": [ 11:56:15 DEBUG "data": { 11:56:15 DEBUG { 11:56:15 DEBUG "data": { 11:56:15 DEBUG "data": { 11:56:15 DEBUG "data": { 11:56:15 DEBUG "instance": "10.142.0.51:9100" 11:56:15 DEBUG { 11:56:15 DEBUG "resultType": "vector", 11:56:15 DEBUG "status": "success", 11:56:15 DEBUG "resultType": "vector", 11:56:15 DEBUG "resultType": "vector", 11:56:15 DEBUG "resultType": "vector", 11:56:15 DEBUG }, 11:56:15 DEBUG "metric": { 11:56:15 DEBUG "result": [ 11:56:15 DEBUG "data": { 11:56:15 DEBUG "result": [ 11:56:15 DEBUG "result": [ 11:56:15 DEBUG "result": [ 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "pod": "ds-idrepo-0" 11:56:15 DEBUG { 11:56:15 DEBUG "resultType": "vector", 11:56:15 DEBUG { 11:56:15 DEBUG { 11:56:15 DEBUG { 11:56:15 DEBUG 1735905375, 11:56:15 DEBUG }, 11:56:15 DEBUG "metric": { 11:56:15 DEBUG "result": [ 11:56:15 DEBUG "metric": { 11:56:15 DEBUG "metric": { 11:56:15 DEBUG "metric": { 11:56:15 DEBUG "0.0005666666666722146" 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "instance": "10.142.0.11:9100" 11:56:15 DEBUG { 11:56:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:56:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:56:15 DEBUG "instance": "10.142.0.11:9100" 11:56:15 DEBUG ] 11:56:15 DEBUG 11:56:15 DEBUG }, 11:56:15 DEBUG "metric": { 11:56:15 DEBUG }, 11:56:15 DEBUG }, 11:56:15 DEBUG }, 11:56:15 DEBUG } 11:56:15 DEBUG 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "value": [ 11:56:15 DEBUG ] 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG }, 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG } 11:56:15 DEBUG [...] 11:56:15 DEBUG 11:56:15 DEBUG "value": [ 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG } 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG [...] 11:56:15 DEBUG 11:56:15 DEBUG [...] 11:56:15 DEBUG [...] 11:56:15 DEBUG [...] 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:56:15 DEBUG 11:56:15 DEBUG [...] 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG }, 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "instance": "10.142.0.51:9100" 11:56:15 DEBUG 11:56:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:56:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:56:15 DEBUG "instance": "10.142.0.51:9100" 11:56:15 DEBUG 1735905375, 11:56:15 DEBUG }, 11:56:15 DEBUG 11:56:15 DEBUG }, 11:56:15 DEBUG }, 11:56:15 DEBUG }, 11:56:15 DEBUG "758542336" 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "pod": "ds-cts-0" 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "value": [ 11:56:15 DEBUG ] 11:56:15 DEBUG 1735905375, 11:56:15 DEBUG }, 11:56:15 DEBUG 1735905375, 11:56:15 DEBUG 1735905375, 11:56:15 DEBUG 1735905375, 11:56:15 DEBUG } 11:56:15 DEBUG "34542.933333333334" 11:56:15 DEBUG "value": [ 11:56:15 DEBUG "14990163968" 11:56:15 DEBUG "0.03480423396147851" 11:56:15 DEBUG "1014356.7000000001" 11:56:15 DEBUG ] 11:56:15 DEBUG ] 11:56:15 DEBUG 1735905375, 11:56:15 DEBUG ] 11:56:15 DEBUG ] 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 DEBUG "0" 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 DEBUG ] 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:15 DEBUG } 11:56:18 INFO 11:56:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:56:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:56:18 INFO [loop_until]: OK (rc = 0) 11:56:18 DEBUG --- stdout --- 11:56:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 129m 0% 3889Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 93m 0% 19058Mi 67% 11:56:18 DEBUG --- stderr --- 11:56:18 DEBUG 11:56:22 INFO 11:56:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:56:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:56:22 INFO [loop_until]: OK (rc = 0) 11:56:22 DEBUG --- stdout --- 11:56:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 35m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:56:22 DEBUG --- stderr --- 11:56:22 DEBUG 11:57:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905433" 11:57:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905433" 11:57:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905433" 11:57:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905433" 11:57:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905433" 11:57:13 INFO [http_cmd]: http status code OK 11:57:13 INFO [http_cmd]: http status code OK 11:57:13 INFO [http_cmd]: http status code OK 11:57:13 DEBUG --- status code --- 11:57:13 ERROR [http_cmd]: ERROR 11:57:13 INFO [http_cmd]: http status code OK 11:57:13 DEBUG --- status code --- 11:57:13 DEBUG --- status code --- 11:57:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:13 ERROR --- status code --- 11:57:13 DEBUG --- status code --- 11:57:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:13 DEBUG --- http response --- 11:57:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:57:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:13 DEBUG --- http response --- 11:57:13 DEBUG --- http response --- 11:57:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:13 ERROR --- http response --- 11:57:13 DEBUG --- http response --- 11:57:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:57:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905434" 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 DEBUG --- status code --- 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:14 INFO [http_cmd]: http status code OK 11:57:14 DEBUG --- status code --- 11:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:14 DEBUG --- http response --- 11:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905435" 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735905435, "269.4470940367727" ] } ] } } 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905435" 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905435" 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735905435, "0" ] } ] } } 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:57:15 DEBUG ----- output ----- 11:57:15 DEBUG { 11:57:15 DEBUG "status": "success", 11:57:15 DEBUG "data": { 11:57:15 DEBUG "resultType": "vector", 11:57:15 DEBUG "result": [ 11:57:15 DEBUG { 11:57:15 DEBUG "metric": { 11:57:15 DEBUG "instance": "10.142.0.11:9100" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG [...] 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG "instance": "10.142.0.51:9100" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 1735905435, 11:57:15 DEBUG "1003683.4666666668" 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905435" 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:57:15 DEBUG ----- output ----- 11:57:15 DEBUG { 11:57:15 DEBUG "status": "success", 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905435" 11:57:15 DEBUG "data": { 11:57:15 DEBUG "resultType": "vector", 11:57:15 DEBUG "result": [ 11:57:15 DEBUG { 11:57:15 DEBUG "metric": { 11:57:15 DEBUG "instance": "10.142.0.11:9100" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG [...] 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG "instance": "10.142.0.51:9100" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 1735905435, 11:57:15 DEBUG "0.001000000000000038" 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905435" 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905435" 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905435" 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905435" 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905435" 11:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905435" 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:57:15 DEBUG ----- output ----- 11:57:15 DEBUG { 11:57:15 DEBUG "status": "success", 11:57:15 DEBUG "data": { 11:57:15 DEBUG "resultType": "vector", 11:57:15 DEBUG "result": [ 11:57:15 DEBUG { 11:57:15 DEBUG "metric": { 11:57:15 DEBUG "instance": "10.142.0.11:9100" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG [...] 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG "instance": "10.142.0.51:9100" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 1735905435, 11:57:15 DEBUG "0.0012333333333316432" 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:57:15 DEBUG ----- output ----- 11:57:15 DEBUG { 11:57:15 DEBUG "status": "success", 11:57:15 DEBUG "data": { 11:57:15 DEBUG "resultType": "vector", 11:57:15 DEBUG "result": [ 11:57:15 DEBUG { 11:57:15 DEBUG "metric": { 11:57:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG [...] 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 1735905435, 11:57:15 DEBUG "0.03394015880210077" 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG --- http response --- 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG --- status code --- 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 INFO [http_cmd]: http status code OK 11:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG --- status code --- 11:57:15 DEBUG ----- output ----- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:57:15 DEBUG { 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG --- http response --- 11:57:15 DEBUG "status": "success", 11:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:57:15 DEBUG ----- output ----- 11:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:57:15 DEBUG "data": { 11:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:57:15 DEBUG ----- output ----- 11:57:15 DEBUG { 11:57:15 DEBUG ----- output ----- 11:57:15 DEBUG "resultType": "vector", 11:57:15 DEBUG ----- output ----- 11:57:15 DEBUG { 11:57:15 DEBUG "status": "success", 11:57:15 DEBUG { 11:57:15 DEBUG "result": [ 11:57:15 DEBUG { 11:57:15 DEBUG "status": "success", 11:57:15 DEBUG "data": { 11:57:15 DEBUG "status": "success", 11:57:15 DEBUG { 11:57:15 DEBUG "status": "success", 11:57:15 DEBUG "data": { 11:57:15 DEBUG "resultType": "vector", 11:57:15 DEBUG "data": { 11:57:15 DEBUG "metric": { 11:57:15 DEBUG "data": { 11:57:15 DEBUG "resultType": "vector", 11:57:15 DEBUG "result": [ 11:57:15 DEBUG "resultType": "vector", 11:57:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:57:15 DEBUG "resultType": "vector", 11:57:15 DEBUG "result": [ 11:57:15 DEBUG { 11:57:15 DEBUG "result": [ 11:57:15 DEBUG }, 11:57:15 DEBUG "result": [ 11:57:15 DEBUG { 11:57:15 DEBUG "metric": { 11:57:15 DEBUG { 11:57:15 DEBUG "value": [ 11:57:15 DEBUG { 11:57:15 DEBUG "metric": { 11:57:15 DEBUG "instance": "10.142.0.11:9100" 11:57:15 DEBUG "metric": { 11:57:15 DEBUG 11:57:15 DEBUG "metric": { 11:57:15 DEBUG "instance": "10.142.0.11:9100" 11:57:15 DEBUG }, 11:57:15 DEBUG "pod": "ds-idrepo-0" 11:57:15 DEBUG 11:57:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG }, 11:57:15 DEBUG 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 11:57:15 DEBUG "value": [ 11:57:15 DEBUG [...] 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG [...] 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG [...] 11:57:15 DEBUG 11:57:15 DEBUG [...] 11:57:15 DEBUG "pod": "ds-cts-0" 11:57:15 DEBUG [...] 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG }, 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 11:57:15 DEBUG 11:57:15 DEBUG "instance": "10.142.0.51:9100" 11:57:15 DEBUG 11:57:15 DEBUG 1735905435, 11:57:15 DEBUG 11:57:15 DEBUG "instance": "10.142.0.51:9100" 11:57:15 DEBUG }, 11:57:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:57:15 DEBUG "0" 11:57:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG }, 11:57:15 DEBUG ] 11:57:15 DEBUG }, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 1735905435, 11:57:15 DEBUG "value": [ 11:57:15 DEBUG } 11:57:15 DEBUG "value": [ 11:57:15 DEBUG 1735905435, 11:57:15 DEBUG "984398.2999999999" 11:57:15 DEBUG 1735905435, 11:57:15 DEBUG ] 11:57:15 DEBUG 1735905435, 11:57:15 DEBUG "32768" 11:57:15 DEBUG ] 11:57:15 DEBUG "758542336" 11:57:15 DEBUG } 11:57:15 DEBUG "14989955072" 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG ] 11:57:15 DEBUG ] 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:15 DEBUG } 11:57:18 INFO 11:57:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:57:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:57:18 INFO [loop_until]: OK (rc = 0) 11:57:18 DEBUG --- stdout --- 11:57:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 129m 0% 3893Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 568m 1% 19053Mi 66% 11:57:18 DEBUG --- stderr --- 11:57:18 DEBUG 11:57:22 INFO 11:57:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:57:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:57:22 INFO [loop_until]: OK (rc = 0) 11:57:22 DEBUG --- stdout --- 11:57:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 392m 13478Mi rcs-645c86fdd6-j4rsk 20m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:57:22 DEBUG --- stderr --- 11:57:22 DEBUG 11:58:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905493" 11:58:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905493" 11:58:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905493" 11:58:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905493" 11:58:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905493" 11:58:13 INFO [http_cmd]: http status code OK 11:58:13 DEBUG --- status code --- 11:58:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:13 DEBUG --- http response --- 11:58:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:13 INFO [http_cmd]: http status code OK 11:58:13 DEBUG --- status code --- 11:58:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:13 DEBUG --- http response --- 11:58:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:13 INFO [http_cmd]: http status code OK 11:58:13 DEBUG --- status code --- 11:58:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:13 DEBUG --- http response --- 11:58:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:13 ERROR [http_cmd]: ERROR 11:58:13 ERROR --- status code --- 11:58:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:58:13 ERROR --- http response --- 11:58:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:58:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:58:13 INFO [http_cmd]: http status code OK 11:58:13 DEBUG --- status code --- 11:58:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:13 DEBUG --- http response --- 11:58:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905494" 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:14 INFO [http_cmd]: http status code OK 11:58:14 DEBUG --- status code --- 11:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:14 DEBUG --- http response --- 11:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905495" 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG { 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "data": { 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "result": [ 11:58:15 DEBUG { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "pod": "ds-cts-0" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG [...] 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG "0" 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905495" 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905495" 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG { 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "data": { 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "result": [ 11:58:15 DEBUG { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "pod": "ds-cts-0" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG [...] 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG "0" 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG { 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "data": { 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "result": [ 11:58:15 DEBUG { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "instance": "10.142.0.11:9100" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG [...] 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG "instance": "10.142.0.51:9100" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG "1814558.8999999997" 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905495" 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG { 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "data": { 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "result": [ 11:58:15 DEBUG { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "instance": "10.142.0.11:9100" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG [...] 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG "instance": "10.142.0.51:9100" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG "0" 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905495" 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905495" 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG { 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "data": { 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "result": [ 11:58:15 DEBUG { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "instance": "10.142.0.11:9100" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG [...] 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG "instance": "10.142.0.51:9100" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG "0.00043333333333445503" 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG { 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "data": { 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "result": [ 11:58:15 DEBUG { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG [...] 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG "0.1805009226875292" 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905495" 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905495" 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905495" 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905495" 11:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905495" 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG { 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "data": { 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "result": [ 11:58:15 DEBUG { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "pod": "ds-cts-0" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG [...] 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG "pod": "ds-idrepo-0" 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG "0" 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG { 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "data": { 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "result": [ 11:58:15 DEBUG { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "instance": "10.142.0.11:9100" 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG }, 11:58:15 INFO [http_cmd]: http status code OK 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG "value": [ 11:58:15 DEBUG --- status code --- 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG 11:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG 11:58:15 DEBUG --- http response --- 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:58:15 DEBUG 11:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG [...] 11:58:15 DEBUG ----- output ----- 11:58:15 DEBUG { 11:58:15 DEBUG { 11:58:15 DEBUG 11:58:15 DEBUG { 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG 11:58:15 DEBUG "status": "success", 11:58:15 DEBUG "data": { 11:58:15 DEBUG "data": { 11:58:15 DEBUG 11:58:15 DEBUG "data": { 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "instance": "10.142.0.51:9100" 11:58:15 DEBUG "resultType": "vector", 11:58:15 DEBUG "result": [ 11:58:15 DEBUG "result": [ 11:58:15 DEBUG }, 11:58:15 DEBUG "result": [ 11:58:15 DEBUG { 11:58:15 DEBUG { 11:58:15 DEBUG "value": [ 11:58:15 DEBUG { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "metric": { 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG "metric": { 11:58:15 DEBUG "instance": "10.142.0.11:9100" 11:58:15 DEBUG "pod": "ds-idrepo-0" 11:58:15 DEBUG "20070.399999999998" 11:58:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:58:15 DEBUG }, 11:58:15 DEBUG }, 11:58:15 DEBUG ] 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG "value": [ 11:58:15 DEBUG } 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG ] 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG } 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG } 11:58:15 DEBUG 11:58:15 DEBUG [...] 11:58:15 DEBUG [...] 11:58:15 DEBUG [...] 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG 11:58:15 DEBUG "instance": "10.142.0.51:9100" 11:58:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:58:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:58:15 DEBUG }, 11:58:15 DEBUG }, 11:58:15 DEBUG }, 11:58:15 DEBUG "value": [ 11:58:15 DEBUG "value": [ 11:58:15 DEBUG "value": [ 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG 1735905495, 11:58:15 DEBUG "1794608.6333333326" 11:58:15 DEBUG "758542336" 11:58:15 DEBUG "14985412608" 11:58:15 DEBUG ] 11:58:15 DEBUG ] 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 DEBUG ] 11:58:15 DEBUG ] 11:58:15 DEBUG ] 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:15 DEBUG } 11:58:18 INFO 11:58:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:58:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:58:18 INFO [loop_until]: OK (rc = 0) 11:58:18 DEBUG --- stdout --- 11:58:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 173m 1% 3869Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 139m 0% 19057Mi 67% 11:58:18 DEBUG --- stderr --- 11:58:18 DEBUG 11:58:22 INFO 11:58:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:58:22 INFO [loop_until]: OK (rc = 0) 11:58:22 DEBUG --- stdout --- 11:58:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 85m 13478Mi rcs-645c86fdd6-j4rsk 124m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:58:22 DEBUG --- stderr --- 11:58:22 DEBUG 11:59:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905553" 11:59:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905553" 11:59:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905553" 11:59:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905553" 11:59:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905553" 11:59:13 INFO [http_cmd]: http status code OK 11:59:13 DEBUG --- status code --- 11:59:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:13 DEBUG --- http response --- 11:59:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:13 INFO [http_cmd]: http status code OK 11:59:13 DEBUG --- status code --- 11:59:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:13 DEBUG --- http response --- 11:59:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:13 INFO [http_cmd]: http status code OK 11:59:13 DEBUG --- status code --- 11:59:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:13 DEBUG --- http response --- 11:59:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:13 INFO [http_cmd]: http status code OK 11:59:13 DEBUG --- status code --- 11:59:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:13 DEBUG --- http response --- 11:59:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:13 ERROR [http_cmd]: ERROR 11:59:13 ERROR --- status code --- 11:59:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 11:59:13 ERROR --- http response --- 11:59:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 11:59:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905554" 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: http status code OK 11:59:14 DEBUG --- status code --- 11:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:14 DEBUG --- http response --- 11:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905554" 11:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905554" 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905555" 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905555" 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "pod": "ds-idrepo-0" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "379.61075069508803" 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905555" 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905555" 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "pod": "ds-idrepo-0" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "0" 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "instance": "10.142.0.11:9100" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG "instance": "10.142.0.51:9100" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "2379634.9" 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905555" 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "instance": "10.142.0.11:9100" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG "instance": "10.142.0.51:9100" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "0.0006666666666674814" 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905555" 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905555" 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "instance": "10.142.0.11:9100" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG "instance": "10.142.0.51:9100" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "0.000466666666664158" 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "0.0432115194062141" 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905555" 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905555" 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905555" 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905555" 11:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905555" 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "pod": "ds-idrepo-0" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "0.0008855900642056035" 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "instance": "10.142.0.11:9100" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG "instance": "10.142.0.51:9100" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "2360399.366666667" 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 INFO [http_cmd]: http status code OK 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- status code --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:59:15 DEBUG --- http response --- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG ----- output ----- 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "status": "success", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "data": { 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG "resultType": "vector", 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "result": [ 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG { 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "instance": "10.142.0.11:9100" 11:59:15 DEBUG "metric": { 11:59:15 DEBUG "pod": "ds-idrepo-0" 11:59:15 DEBUG }, 11:59:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG [...] 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG 11:59:15 DEBUG "instance": "10.142.0.51:9100" 11:59:15 DEBUG 11:59:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 11:59:15 DEBUG }, 11:59:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG }, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "value": [ 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "28808.533333333333" 11:59:15 DEBUG 1735905555, 11:59:15 DEBUG "758542336" 11:59:15 DEBUG ] 11:59:15 DEBUG "14986235904" 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG ] 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:15 DEBUG } 11:59:18 INFO 11:59:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 11:59:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:59:18 INFO [loop_until]: OK (rc = 0) 11:59:18 DEBUG --- stdout --- 11:59:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 221m 1% 3859Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 179m 0% 19060Mi 67% 11:59:18 DEBUG --- stderr --- 11:59:18 DEBUG 11:59:22 INFO 11:59:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 11:59:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 11:59:22 INFO [loop_until]: OK (rc = 0) 11:59:22 DEBUG --- stdout --- 11:59:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 112m 13478Mi rcs-645c86fdd6-j4rsk 117m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 11:59:22 DEBUG --- stderr --- 11:59:22 DEBUG 12:00:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905613" 12:00:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905613" 12:00:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905613" 12:00:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905613" 12:00:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905613" 12:00:13 INFO [http_cmd]: http status code OK 12:00:13 ERROR [http_cmd]: ERROR 12:00:13 DEBUG --- status code --- 12:00:13 INFO [http_cmd]: http status code OK 12:00:13 ERROR --- status code --- 12:00:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:13 DEBUG --- status code --- 12:00:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:00:13 DEBUG --- http response --- 12:00:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:13 ERROR --- http response --- 12:00:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:13 DEBUG --- http response --- 12:00:13 INFO [http_cmd]: http status code OK 12:00:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:00:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:13 DEBUG --- status code --- 12:00:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:00:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:13 DEBUG --- http response --- 12:00:13 INFO [http_cmd]: http status code OK 12:00:13 DEBUG --- status code --- 12:00:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:13 DEBUG --- http response --- 12:00:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905614" 12:00:14 INFO [http_cmd]: http status code OK 12:00:14 DEBUG --- status code --- 12:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:14 DEBUG --- http response --- 12:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG { 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG "data": { 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG "result": [ 12:00:15 DEBUG { 12:00:15 DEBUG "metric": { 12:00:15 DEBUG "pod": "ds-cts-0" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG [...] 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "147.9661874142042" 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905615" 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG { 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG "data": { 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG "result": [ 12:00:15 DEBUG { 12:00:15 DEBUG "metric": { 12:00:15 DEBUG "pod": "ds-cts-0" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG [...] 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "0" 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG { 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG "data": { 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG "result": [ 12:00:15 DEBUG { 12:00:15 DEBUG "metric": { 12:00:15 DEBUG "instance": "10.142.0.11:9100" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG [...] 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "instance": "10.142.0.51:9100" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "2376693.4666666663" 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905615" 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG { 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG "data": { 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG "result": [ 12:00:15 DEBUG { 12:00:15 DEBUG "metric": { 12:00:15 DEBUG "instance": "10.142.0.11:9100" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG [...] 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "instance": "10.142.0.51:9100" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "0.0003333333333330302" 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905615" 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG { 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG "data": { 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG "result": [ 12:00:15 DEBUG { 12:00:15 DEBUG "metric": { 12:00:15 DEBUG "instance": "10.142.0.11:9100" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG [...] 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "instance": "10.142.0.51:9100" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "0.00046666666666700016" 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG { 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG "data": { 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG "result": [ 12:00:15 DEBUG { 12:00:15 DEBUG "metric": { 12:00:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG [...] 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "0.13450796994175032" 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905615" 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG { 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG "data": { 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG "result": [ 12:00:15 DEBUG { 12:00:15 DEBUG "metric": { 12:00:15 DEBUG "pod": "ds-cts-0" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG [...] 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "0.0009254805852356283" 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905615" 12:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905615" 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG { 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG "data": { 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG "result": [ 12:00:15 DEBUG { 12:00:15 DEBUG "metric": { 12:00:15 DEBUG "instance": "10.142.0.11:9100" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG [...] 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "instance": "10.142.0.51:9100" 12:00:15 DEBUG }, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "2357542.8333333326" 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG { 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG "data": { 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG "result": [ 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:00:15 DEBUG { 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG "metric": { 12:00:15 DEBUG { 12:00:15 DEBUG "status": "success", 12:00:15 INFO [http_cmd]: http status code OK 12:00:15 DEBUG --- status code --- 12:00:15 DEBUG "data": { 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:00:15 DEBUG --- http response --- 12:00:15 DEBUG "result": [ 12:00:15 DEBUG { 12:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:00:15 DEBUG "pod": "ds-idrepo-0" 12:00:15 DEBUG "metric": { 12:00:15 DEBUG ----- output ----- 12:00:15 DEBUG }, 12:00:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:00:15 DEBUG { 12:00:15 DEBUG "value": [ 12:00:15 DEBUG }, 12:00:15 DEBUG "status": "success", 12:00:15 DEBUG 12:00:15 DEBUG "value": [ 12:00:15 DEBUG "data": { 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "resultType": "vector", 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "result": [ 12:00:15 DEBUG [...] 12:00:15 DEBUG 12:00:15 DEBUG { 12:00:15 DEBUG 12:00:15 DEBUG [...] 12:00:15 DEBUG "metric": { 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG "instance": "10.142.0.11:9100" 12:00:15 DEBUG 12:00:15 DEBUG 12:00:15 DEBUG }, 12:00:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:00:15 DEBUG 12:00:15 DEBUG "value": [ 12:00:15 DEBUG }, 12:00:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:00:15 DEBUG 12:00:15 DEBUG "value": [ 12:00:15 DEBUG }, 12:00:15 DEBUG 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 12:00:15 DEBUG "758542336" 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG [...] 12:00:15 DEBUG ] 12:00:15 DEBUG "14991691776" 12:00:15 DEBUG 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG "instance": "10.142.0.51:9100" 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:15 DEBUG }, 12:00:15 DEBUG } 12:00:15 DEBUG "value": [ 12:00:15 DEBUG 1735905615, 12:00:15 DEBUG "28262.4" 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG ] 12:00:15 DEBUG } 12:00:15 DEBUG } 12:00:18 INFO 12:00:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:00:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:00:18 INFO [loop_until]: OK (rc = 0) 12:00:18 DEBUG --- stdout --- 12:00:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 226m 1% 3839Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 189m 0% 19059Mi 67% 12:00:18 DEBUG --- stderr --- 12:00:18 DEBUG 12:00:22 INFO 12:00:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:00:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:00:22 INFO [loop_until]: OK (rc = 0) 12:00:22 DEBUG --- stdout --- 12:00:22 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 155m 13479Mi rcs-645c86fdd6-j4rsk 119m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:00:22 DEBUG --- stderr --- 12:00:22 DEBUG 12:01:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905673" 12:01:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905673" 12:01:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905673" 12:01:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905673" 12:01:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905673" 12:01:13 INFO [http_cmd]: http status code OK 12:01:13 DEBUG --- status code --- 12:01:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:13 DEBUG --- http response --- 12:01:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:13 INFO [http_cmd]: http status code OK 12:01:13 INFO [http_cmd]: http status code OK 12:01:13 DEBUG --- status code --- 12:01:13 DEBUG --- status code --- 12:01:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:13 DEBUG --- http response --- 12:01:13 DEBUG --- http response --- 12:01:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:13 ERROR [http_cmd]: ERROR 12:01:13 ERROR --- status code --- 12:01:13 INFO [http_cmd]: http status code OK 12:01:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:01:13 DEBUG --- status code --- 12:01:13 ERROR --- http response --- 12:01:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:01:13 DEBUG --- http response --- 12:01:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:01:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905674" 12:01:14 INFO [http_cmd]: http status code OK 12:01:14 DEBUG --- status code --- 12:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:14 DEBUG --- http response --- 12:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905674" 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG { 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "data": { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG { 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "pod": "ds-cts-0" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "pod": "ds-idrepo-0" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG "1309.54171186041" 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG { 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "data": { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG { 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "pod": "ds-cts-0" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "pod": "ds-idrepo-0" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG "236154.0220388273" 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905675" 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG { 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "data": { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG { 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "instance": "10.142.0.11:9100" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "instance": "10.142.0.51:9100" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG "1233505.0666666667" 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905675" 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG { 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "data": { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG { 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "instance": "10.142.0.11:9100" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "instance": "10.142.0.51:9100" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG "0.001000000000000038" 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905675" 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG { 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "data": { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG { 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "instance": "10.142.0.11:9100" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "instance": "10.142.0.51:9100" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG "0.0014333333333297561" 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG { 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "data": { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG { 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG "0.12528011468562952" 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905675" 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG { 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "data": { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG { 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "pod": "ds-cts-0" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "pod": "ds-idrepo-0" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG "0.011894098031414549" 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG { 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "data": { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG { 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "instance": "10.142.0.11:9100" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "instance": "10.142.0.51:9100" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG "1214532.4666666663" 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905675" 12:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905675" 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG { 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG "data": { 12:01:15 DEBUG { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG "data": { 12:01:15 DEBUG { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "result": [ 12:01:15 DEBUG "instance": "10.142.0.11:9100" 12:01:15 DEBUG { 12:01:15 DEBUG }, 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "value": [ 12:01:15 DEBUG "pod": "ds-idrepo-0" 12:01:15 DEBUG 12:01:15 DEBUG }, 12:01:15 DEBUG 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "instance": "10.142.0.51:9100" 12:01:15 DEBUG 12:01:15 DEBUG }, 12:01:15 DEBUG 12:01:15 DEBUG "value": [ 12:01:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG }, 12:01:15 DEBUG "318668.8" 12:01:15 DEBUG "value": [ 12:01:15 DEBUG ] 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG } 12:01:15 DEBUG "758542336" 12:01:15 DEBUG ] 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 INFO [http_cmd]: http status code OK 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:15 DEBUG --- status code --- 12:01:15 DEBUG ] 12:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:01:15 DEBUG } 12:01:15 DEBUG --- http response --- 12:01:15 DEBUG } 12:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:01:15 DEBUG ----- output ----- 12:01:15 DEBUG { 12:01:15 DEBUG "status": "success", 12:01:15 DEBUG "data": { 12:01:15 DEBUG "resultType": "vector", 12:01:15 DEBUG "result": [ 12:01:15 DEBUG { 12:01:15 DEBUG "metric": { 12:01:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG [...] 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG 12:01:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:01:15 DEBUG }, 12:01:15 DEBUG "value": [ 12:01:15 DEBUG 1735905675, 12:01:15 DEBUG "14991667200" 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG ] 12:01:15 DEBUG } 12:01:15 DEBUG } 12:01:18 INFO 12:01:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:01:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:01:18 INFO [loop_until]: OK (rc = 0) 12:01:18 DEBUG --- stdout --- 12:01:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 206m 1% 3845Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 199m 0% 19060Mi 67% 12:01:18 DEBUG --- stderr --- 12:01:18 DEBUG 12:01:22 INFO 12:01:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:01:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:01:23 INFO [loop_until]: OK (rc = 0) 12:01:23 DEBUG --- stdout --- 12:01:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 108m 13479Mi rcs-645c86fdd6-j4rsk 84m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:01:23 DEBUG --- stderr --- 12:01:23 DEBUG 12:02:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905733" 12:02:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905733" 12:02:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905733" 12:02:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905733" 12:02:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905733" 12:02:13 INFO [http_cmd]: http status code OK 12:02:13 INFO [http_cmd]: http status code OK 12:02:13 INFO [http_cmd]: http status code OK 12:02:13 DEBUG --- status code --- 12:02:13 ERROR [http_cmd]: ERROR 12:02:13 INFO [http_cmd]: http status code OK 12:02:13 DEBUG --- status code --- 12:02:13 DEBUG --- status code --- 12:02:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:13 ERROR --- status code --- 12:02:13 DEBUG --- status code --- 12:02:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:13 DEBUG --- http response --- 12:02:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:02:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:13 DEBUG --- http response --- 12:02:13 DEBUG --- http response --- 12:02:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:13 ERROR --- http response --- 12:02:13 DEBUG --- http response --- 12:02:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:02:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905734" 12:02:14 INFO [http_cmd]: http status code OK 12:02:14 DEBUG --- status code --- 12:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:14 DEBUG --- http response --- 12:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735905735, "489.11842851598647" ] } ] } } 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905735" 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735905735, "0" ] } ] } } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:02:15 DEBUG ----- output ----- 12:02:15 DEBUG { 12:02:15 DEBUG "status": "success", 12:02:15 DEBUG "data": { 12:02:15 DEBUG "resultType": "vector", 12:02:15 DEBUG "result": [ 12:02:15 DEBUG { 12:02:15 DEBUG "metric": { 12:02:15 DEBUG "instance": "10.142.0.11:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG [...] 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG "instance": "10.142.0.51:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 1735905735, 12:02:15 DEBUG "1228439.066666667" 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905735" 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:02:15 DEBUG ----- output ----- 12:02:15 DEBUG { 12:02:15 DEBUG "status": "success", 12:02:15 DEBUG "data": { 12:02:15 DEBUG "resultType": "vector", 12:02:15 DEBUG "result": [ 12:02:15 DEBUG { 12:02:15 DEBUG "metric": { 12:02:15 DEBUG "instance": "10.142.0.11:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG [...] 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG "instance": "10.142.0.51:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 1735905735, 12:02:15 DEBUG "0.001000000000000038" 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905735" 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:02:15 DEBUG ----- output ----- 12:02:15 DEBUG { 12:02:15 DEBUG "status": "success", 12:02:15 DEBUG "data": { 12:02:15 DEBUG "resultType": "vector", 12:02:15 DEBUG "result": [ 12:02:15 DEBUG { 12:02:15 DEBUG "metric": { 12:02:15 DEBUG "instance": "10.142.0.11:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG [...] 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG "instance": "10.142.0.51:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 1735905735, 12:02:15 DEBUG "0.0011999999999981507" 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:02:15 DEBUG ----- output ----- 12:02:15 DEBUG { 12:02:15 DEBUG "status": "success", 12:02:15 DEBUG "data": { 12:02:15 DEBUG "resultType": "vector", 12:02:15 DEBUG "result": [ 12:02:15 DEBUG { 12:02:15 DEBUG "metric": { 12:02:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG [...] 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 1735905735, 12:02:15 DEBUG "0.19853936657778354" 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905735" 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:02:15 DEBUG ----- output ----- 12:02:15 DEBUG { 12:02:15 DEBUG "status": "success", 12:02:15 DEBUG "data": { 12:02:15 DEBUG "resultType": "vector", 12:02:15 DEBUG "result": [ 12:02:15 DEBUG { 12:02:15 DEBUG "metric": { 12:02:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG [...] 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG "pod": "ds-cts-0" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 1735905735, 12:02:15 DEBUG "0" 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:02:15 DEBUG ----- output ----- 12:02:15 DEBUG { 12:02:15 DEBUG "status": "success", 12:02:15 DEBUG "data": { 12:02:15 DEBUG "resultType": "vector", 12:02:15 DEBUG "result": [ 12:02:15 DEBUG { 12:02:15 DEBUG "metric": { 12:02:15 DEBUG "instance": "10.142.0.11:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG [...] 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG "instance": "10.142.0.51:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 1735905735, 12:02:15 DEBUG "1209405.0333333332" 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905735" 12:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905735" 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:02:15 DEBUG ----- output ----- 12:02:15 DEBUG { 12:02:15 DEBUG "status": "success", 12:02:15 DEBUG "data": { 12:02:15 DEBUG "resultType": "vector", 12:02:15 DEBUG "result": [ 12:02:15 DEBUG { 12:02:15 DEBUG "metric": { 12:02:15 DEBUG "instance": "10.142.0.11:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG [...] 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG "instance": "10.142.0.51:9100" 12:02:15 DEBUG }, 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 1735905735, 12:02:15 DEBUG "28535.466666666667" 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 INFO [http_cmd]: http status code OK 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG --- status code --- 12:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:02:15 DEBUG ----- output ----- 12:02:15 DEBUG --- http response --- 12:02:15 DEBUG { 12:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:02:15 DEBUG "status": "success", 12:02:15 DEBUG ----- output ----- 12:02:15 DEBUG "data": { 12:02:15 DEBUG { 12:02:15 DEBUG "resultType": "vector", 12:02:15 DEBUG "status": "success", 12:02:15 DEBUG "result": [ 12:02:15 DEBUG "data": { 12:02:15 DEBUG { 12:02:15 DEBUG "resultType": "vector", 12:02:15 DEBUG "metric": { 12:02:15 DEBUG "result": [ 12:02:15 DEBUG "pod": "ds-idrepo-0" 12:02:15 DEBUG { 12:02:15 DEBUG }, 12:02:15 DEBUG "metric": { 12:02:15 DEBUG "value": [ 12:02:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:02:15 DEBUG 12:02:15 DEBUG }, 12:02:15 DEBUG 12:02:15 DEBUG "value": [ 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG [...] 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG [...] 12:02:15 DEBUG 12:02:15 DEBUG 12:02:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:02:15 DEBUG 12:02:15 DEBUG }, 12:02:15 DEBUG 12:02:15 DEBUG "value": [ 12:02:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:02:15 DEBUG 1735905735, 12:02:15 DEBUG }, 12:02:15 DEBUG "758542336" 12:02:15 DEBUG "value": [ 12:02:15 DEBUG ] 12:02:15 DEBUG 1735905735, 12:02:15 DEBUG } 12:02:15 DEBUG "14991814656" 12:02:15 DEBUG ] 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:15 DEBUG ] 12:02:15 DEBUG } 12:02:15 DEBUG } 12:02:18 INFO 12:02:18 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:02:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:02:19 INFO [loop_until]: OK (rc = 0) 12:02:19 DEBUG --- stdout --- 12:02:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 179m 1% 3806Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 129m 0% 19059Mi 67% 12:02:19 DEBUG --- stderr --- 12:02:19 DEBUG 12:02:23 INFO 12:02:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:02:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:02:23 INFO [loop_until]: OK (rc = 0) 12:02:23 DEBUG --- stdout --- 12:02:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 51m 13479Mi rcs-645c86fdd6-j4rsk 51m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:02:23 DEBUG --- stderr --- 12:02:23 DEBUG 12:03:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905793" 12:03:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905793" 12:03:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905793" 12:03:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905793" 12:03:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905793" 12:03:13 INFO [http_cmd]: http status code OK 12:03:13 DEBUG --- status code --- 12:03:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:13 DEBUG --- http response --- 12:03:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:13 INFO [http_cmd]: http status code OK 12:03:13 INFO [http_cmd]: http status code OK 12:03:13 INFO [http_cmd]: http status code OK 12:03:13 DEBUG --- status code --- 12:03:13 ERROR [http_cmd]: ERROR 12:03:13 DEBUG --- status code --- 12:03:13 DEBUG --- status code --- 12:03:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:13 ERROR --- status code --- 12:03:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:13 DEBUG --- http response --- 12:03:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:03:13 DEBUG --- http response --- 12:03:13 DEBUG --- http response --- 12:03:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:13 ERROR --- http response --- 12:03:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:03:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905794" 12:03:14 INFO [http_cmd]: http status code OK 12:03:14 DEBUG --- status code --- 12:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:14 DEBUG --- http response --- 12:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "440.92001865872476" 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905795" 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "0" 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "instance": "10.142.0.11:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "instance": "10.142.0.51:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "774249.0333333331" 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905795" 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "instance": "10.142.0.11:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "instance": "10.142.0.51:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "0.0003333333333330302" 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905795" 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "instance": "10.142.0.11:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "instance": "10.142.0.51:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "0.00046666666666700016" 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "0.09869633862250565" 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905795" 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "0.0004925903333417392" 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "instance": "10.142.0.11:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "instance": "10.142.0.51:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "755317.2666666668" 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905795" 12:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905795" 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "instance": "10.142.0.11:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "instance": "10.142.0.51:9100" 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "23483.73333333333" 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 INFO [http_cmd]: http status code OK 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG --- status code --- 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG --- http response --- 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG ----- output ----- 12:03:15 DEBUG { 12:03:15 DEBUG { 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "status": "success", 12:03:15 DEBUG "data": { 12:03:15 DEBUG "data": { 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "resultType": "vector", 12:03:15 DEBUG "result": [ 12:03:15 DEBUG "result": [ 12:03:15 DEBUG { 12:03:15 DEBUG { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "metric": { 12:03:15 DEBUG "pod": "ds-idrepo-0" 12:03:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:03:15 DEBUG }, 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG [...] 12:03:15 DEBUG [...] 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG 12:03:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:03:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:03:15 DEBUG }, 12:03:15 DEBUG }, 12:03:15 DEBUG "value": [ 12:03:15 DEBUG "value": [ 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG 1735905795, 12:03:15 DEBUG "758542336" 12:03:15 DEBUG "14989324288" 12:03:15 DEBUG ] 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 DEBUG ] 12:03:15 DEBUG ] 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:15 DEBUG } 12:03:19 INFO 12:03:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:03:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:03:19 INFO [loop_until]: OK (rc = 0) 12:03:19 DEBUG --- stdout --- 12:03:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 178m 1% 3821Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 718m 2% 19059Mi 67% 12:03:19 DEBUG --- stderr --- 12:03:19 DEBUG 12:03:23 INFO 12:03:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:03:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:03:23 INFO [loop_until]: OK (rc = 0) 12:03:23 DEBUG --- stdout --- 12:03:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 641m 13478Mi rcs-645c86fdd6-j4rsk 68m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:03:23 DEBUG --- stderr --- 12:03:23 DEBUG 12:04:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905853" 12:04:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905853" 12:04:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905853" 12:04:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905853" 12:04:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905853" 12:04:13 INFO [http_cmd]: http status code OK 12:04:13 DEBUG --- status code --- 12:04:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:13 DEBUG --- http response --- 12:04:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:13 INFO [http_cmd]: http status code OK 12:04:13 DEBUG --- status code --- 12:04:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:13 DEBUG --- http response --- 12:04:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:13 ERROR [http_cmd]: ERROR 12:04:13 ERROR --- status code --- 12:04:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:04:13 ERROR --- http response --- 12:04:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:04:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:04:13 INFO [http_cmd]: http status code OK 12:04:13 DEBUG --- status code --- 12:04:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:13 DEBUG --- http response --- 12:04:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:13 INFO [http_cmd]: http status code OK 12:04:13 DEBUG --- status code --- 12:04:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:13 DEBUG --- http response --- 12:04:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905854" 12:04:14 DEBUG --- status code --- 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905854" 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905854" 12:04:14 INFO [http_cmd]: http status code OK 12:04:14 DEBUG --- status code --- 12:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:14 DEBUG --- http response --- 12:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "pod": "ds-cts-0" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "590.184720901415" 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905855" 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "pod": "ds-cts-0" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "0" 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "instance": "10.142.0.11:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "instance": "10.142.0.51:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "92606.10000000002" 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905855" 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "instance": "10.142.0.11:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "instance": "10.142.0.51:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "0.0003333333333330302" 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905855" 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "instance": "10.142.0.11:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "instance": "10.142.0.51:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "0.00043333333333066547" 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "0.6632792264662805" 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905855" 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "pod": "ds-cts-0" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "0.002669317887089044" 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "instance": "10.142.0.11:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "instance": "10.142.0.51:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "72763.6" 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905855" 12:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905855" 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "instance": "10.142.0.11:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "instance": "10.142.0.51:9100" 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "22118.399999999998" 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 INFO [http_cmd]: http status code OK 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG --- status code --- 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG --- http response --- 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:04:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG ----- output ----- 12:04:15 DEBUG { 12:04:15 DEBUG { 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "status": "success", 12:04:15 DEBUG "data": { 12:04:15 DEBUG "data": { 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "resultType": "vector", 12:04:15 DEBUG "result": [ 12:04:15 DEBUG "result": [ 12:04:15 DEBUG { 12:04:15 DEBUG { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "metric": { 12:04:15 DEBUG "pod": "ds-idrepo-0" 12:04:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:04:15 DEBUG }, 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG [...] 12:04:15 DEBUG [...] 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG 12:04:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:04:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:04:15 DEBUG }, 12:04:15 DEBUG }, 12:04:15 DEBUG "value": [ 12:04:15 DEBUG "value": [ 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG 1735905855, 12:04:15 DEBUG "758542336" 12:04:15 DEBUG "14988673024" 12:04:15 DEBUG ] 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 DEBUG ] 12:04:15 DEBUG ] 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:15 DEBUG } 12:04:19 INFO 12:04:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:04:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:04:19 INFO [loop_until]: OK (rc = 0) 12:04:19 DEBUG --- stdout --- 12:04:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 148m 0% 3851Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 112m 0% 19062Mi 67% 12:04:19 DEBUG --- stderr --- 12:04:19 DEBUG 12:04:23 INFO 12:04:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:04:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:04:23 INFO [loop_until]: OK (rc = 0) 12:04:23 DEBUG --- stdout --- 12:04:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 60m 13479Mi rcs-645c86fdd6-j4rsk 43m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:04:23 DEBUG --- stderr --- 12:04:23 DEBUG 12:05:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905913" 12:05:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905913" 12:05:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905913" 12:05:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905913" 12:05:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905913" 12:05:13 INFO [http_cmd]: http status code OK 12:05:13 ERROR [http_cmd]: ERROR 12:05:13 INFO [http_cmd]: http status code OK 12:05:13 INFO [http_cmd]: http status code OK 12:05:13 INFO [http_cmd]: http status code OK 12:05:13 DEBUG --- status code --- 12:05:13 ERROR --- status code --- 12:05:13 DEBUG --- status code --- 12:05:13 DEBUG --- status code --- 12:05:13 DEBUG --- status code --- 12:05:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:05:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:13 DEBUG --- http response --- 12:05:13 ERROR --- http response --- 12:05:13 DEBUG --- http response --- 12:05:13 DEBUG --- http response --- 12:05:13 DEBUG --- http response --- 12:05:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:05:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905914" 12:05:14 INFO [http_cmd]: http status code OK 12:05:14 DEBUG --- status code --- 12:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:14 DEBUG --- http response --- 12:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG --- status code --- 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905915" 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "pod": "ds-cts-0" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "420.2328576550735" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905915" 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "pod": "ds-cts-0" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "0" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "instance": "10.142.0.11:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "instance": "10.142.0.51:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "90229.33333333333" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905915" 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "instance": "10.142.0.11:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "instance": "10.142.0.51:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "0.0003333333333330302" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905915" 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "instance": "10.142.0.11:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "instance": "10.142.0.51:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "0.0003999999999981204" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "0.0814420611811597" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905915" 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "pod": "ds-cts-0" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "0.0019684051774820042" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "instance": "10.142.0.11:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "instance": "10.142.0.51:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "70407.8" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905915" 12:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905915" 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "instance": "10.142.0.11:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "instance": "10.142.0.51:9100" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "25668.266666666666" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "pod": "ds-idrepo-0" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "758542336" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:15 INFO [http_cmd]: http status code OK 12:05:15 DEBUG --- status code --- 12:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:05:15 DEBUG --- http response --- 12:05:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:05:15 DEBUG ----- output ----- 12:05:15 DEBUG { 12:05:15 DEBUG "status": "success", 12:05:15 DEBUG "data": { 12:05:15 DEBUG "resultType": "vector", 12:05:15 DEBUG "result": [ 12:05:15 DEBUG { 12:05:15 DEBUG "metric": { 12:05:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG [...] 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG 12:05:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:05:15 DEBUG }, 12:05:15 DEBUG "value": [ 12:05:15 DEBUG 1735905915, 12:05:15 DEBUG "14992424960" 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG ] 12:05:15 DEBUG } 12:05:15 DEBUG } 12:05:19 INFO 12:05:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:05:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:05:19 INFO [loop_until]: OK (rc = 0) 12:05:19 DEBUG --- stdout --- 12:05:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 117m 0% 3848Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 77m 0% 19059Mi 67% 12:05:19 DEBUG --- stderr --- 12:05:19 DEBUG 12:05:23 INFO 12:05:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:05:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:05:23 INFO [loop_until]: OK (rc = 0) 12:05:23 DEBUG --- stdout --- 12:05:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13479Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:05:23 DEBUG --- stderr --- 12:05:23 DEBUG 12:06:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905973" 12:06:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905973" 12:06:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735905973" 12:06:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905973" 12:06:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905973" 12:06:13 INFO [http_cmd]: http status code OK 12:06:13 ERROR [http_cmd]: ERROR 12:06:13 INFO [http_cmd]: http status code OK 12:06:13 INFO [http_cmd]: http status code OK 12:06:13 INFO [http_cmd]: http status code OK 12:06:13 DEBUG --- status code --- 12:06:13 ERROR --- status code --- 12:06:13 DEBUG --- status code --- 12:06:13 DEBUG --- status code --- 12:06:13 DEBUG --- status code --- 12:06:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:06:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:13 DEBUG --- http response --- 12:06:13 ERROR --- http response --- 12:06:13 DEBUG --- http response --- 12:06:13 DEBUG --- http response --- 12:06:13 DEBUG --- http response --- 12:06:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:06:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905974" 12:06:14 INFO [http_cmd]: http status code OK 12:06:14 DEBUG --- status code --- 12:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:14 DEBUG --- http response --- 12:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG --- status code --- 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG --- status code --- 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "pod": "ds-cts-0" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "384.4257533344311" 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905975" 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "pod": "ds-cts-0" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "0" 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "instance": "10.142.0.11:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "instance": "10.142.0.51:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "575104.6" 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905975" 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "instance": "10.142.0.11:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "instance": "10.142.0.51:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "0.0006666666666660604" 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735905975" 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "instance": "10.142.0.11:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "instance": "10.142.0.51:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "0.00043333333333066547" 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "0.048195347126637124" 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905975" 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "pod": "ds-cts-0" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "0.0015476515313661118" 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "instance": "10.142.0.11:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "instance": "10.142.0.51:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "555427.9" 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735905975" 12:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735905975" 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "instance": "10.142.0.11:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "instance": "10.142.0.51:9100" 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "12014.933333333334" 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 INFO [http_cmd]: http status code OK 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG --- status code --- 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG --- http response --- 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:06:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG ----- output ----- 12:06:15 DEBUG { 12:06:15 DEBUG { 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "status": "success", 12:06:15 DEBUG "data": { 12:06:15 DEBUG "data": { 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "resultType": "vector", 12:06:15 DEBUG "result": [ 12:06:15 DEBUG "result": [ 12:06:15 DEBUG { 12:06:15 DEBUG { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "metric": { 12:06:15 DEBUG "pod": "ds-idrepo-0" 12:06:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:06:15 DEBUG }, 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG [...] 12:06:15 DEBUG [...] 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG 12:06:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:06:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:06:15 DEBUG }, 12:06:15 DEBUG }, 12:06:15 DEBUG "value": [ 12:06:15 DEBUG "value": [ 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG 1735905975, 12:06:15 DEBUG "758542336" 12:06:15 DEBUG "14991122432" 12:06:15 DEBUG ] 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 DEBUG ] 12:06:15 DEBUG ] 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:15 DEBUG } 12:06:19 INFO 12:06:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:06:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:06:19 INFO [loop_until]: OK (rc = 0) 12:06:19 DEBUG --- stdout --- 12:06:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 119m 0% 3819Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 79m 0% 19052Mi 66% 12:06:19 DEBUG --- stderr --- 12:06:19 DEBUG 12:06:23 INFO 12:06:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:06:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:06:23 INFO [loop_until]: OK (rc = 0) 12:06:23 DEBUG --- stdout --- 12:06:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 62m 13477Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:06:23 DEBUG --- stderr --- 12:06:23 DEBUG 12:07:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906033" 12:07:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906033" 12:07:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906033" 12:07:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906033" 12:07:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906033" 12:07:13 ERROR [http_cmd]: ERROR 12:07:13 INFO [http_cmd]: http status code OK 12:07:13 INFO [http_cmd]: http status code OK 12:07:13 ERROR --- status code --- 12:07:13 INFO [http_cmd]: http status code OK 12:07:13 DEBUG --- status code --- 12:07:13 INFO [http_cmd]: http status code OK 12:07:13 DEBUG --- status code --- 12:07:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:07:13 DEBUG --- status code --- 12:07:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:13 DEBUG --- status code --- 12:07:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:13 ERROR --- http response --- 12:07:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:13 DEBUG --- http response --- 12:07:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:13 DEBUG --- http response --- 12:07:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:07:13 DEBUG --- http response --- 12:07:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:13 DEBUG --- http response --- 12:07:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:07:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906034" 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: http status code OK 12:07:14 DEBUG --- status code --- 12:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:14 DEBUG --- http response --- 12:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906034" 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906035" 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906035" 12:07:15 DEBUG --- http response --- 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "pod": "ds-cts-0" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "812.9134691717386" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906035" 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "pod": "ds-cts-0" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "0" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "instance": "10.142.0.11:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "instance": "10.142.0.51:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "1241258.7333333336" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906035" 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "instance": "10.142.0.11:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "instance": "10.142.0.51:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "0.0010000000000009853" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906035" 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "instance": "10.142.0.11:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "instance": "10.142.0.51:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "0.0011666666666656055" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "0.037174150683383686" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906035" 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "pod": "ds-cts-0" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "0.002733229690386503" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "instance": "10.142.0.11:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "instance": "10.142.0.51:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "1224361.4333333333" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906035" 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "instance": "10.142.0.11:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "instance": "10.142.0.51:9100" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "25804.8" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906035" 12:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906035" 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "pod": "ds-idrepo-0" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "758542336" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:15 INFO [http_cmd]: http status code OK 12:07:15 DEBUG --- status code --- 12:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:07:15 DEBUG --- http response --- 12:07:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:07:15 DEBUG ----- output ----- 12:07:15 DEBUG { 12:07:15 DEBUG "status": "success", 12:07:15 DEBUG "data": { 12:07:15 DEBUG "resultType": "vector", 12:07:15 DEBUG "result": [ 12:07:15 DEBUG { 12:07:15 DEBUG "metric": { 12:07:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG [...] 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG 12:07:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:07:15 DEBUG }, 12:07:15 DEBUG "value": [ 12:07:15 DEBUG 1735906035, 12:07:15 DEBUG "14986395648" 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG ] 12:07:15 DEBUG } 12:07:15 DEBUG } 12:07:19 INFO 12:07:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:07:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:07:19 INFO [loop_until]: OK (rc = 0) 12:07:19 DEBUG --- stdout --- 12:07:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 145m 0% 3837Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 95m 0% 19055Mi 67% 12:07:19 DEBUG --- stderr --- 12:07:19 DEBUG 12:07:23 INFO 12:07:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:07:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:07:23 INFO [loop_until]: OK (rc = 0) 12:07:23 DEBUG --- stdout --- 12:07:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 58m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:07:23 DEBUG --- stderr --- 12:07:23 DEBUG 12:08:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906093" 12:08:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906093" 12:08:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906093" 12:08:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906093" 12:08:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906093" 12:08:13 INFO [http_cmd]: http status code OK 12:08:13 ERROR [http_cmd]: ERROR 12:08:13 DEBUG --- status code --- 12:08:13 ERROR --- status code --- 12:08:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:08:13 DEBUG --- http response --- 12:08:13 ERROR --- http response --- 12:08:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:08:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:08:13 INFO [http_cmd]: http status code OK 12:08:13 DEBUG --- status code --- 12:08:13 INFO [http_cmd]: http status code OK 12:08:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:13 DEBUG --- status code --- 12:08:13 DEBUG --- http response --- 12:08:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:13 DEBUG --- http response --- 12:08:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:13 INFO [http_cmd]: http status code OK 12:08:13 DEBUG --- status code --- 12:08:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:13 DEBUG --- http response --- 12:08:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 DEBUG --- status code --- 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906094" 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:14 INFO [http_cmd]: http status code OK 12:08:14 DEBUG --- status code --- 12:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:14 DEBUG --- http response --- 12:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG --- status code --- 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG --- status code --- 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG { 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "data": { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG { 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "pod": "ds-cts-0" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "pod": "ds-idrepo-0" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG "2292.831403281671" 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906095" 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG { 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "data": { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG { 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "pod": "ds-cts-0" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "pod": "ds-idrepo-0" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG "3473782.877934437" 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG { 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "data": { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG { 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "instance": "10.142.0.11:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "instance": "10.142.0.51:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG "1239795.6666666672" 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906095" 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906095" 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG { 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "data": { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG { 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "instance": "10.142.0.11:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "instance": "10.142.0.51:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG "0.0010000000000009853" 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906095" 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG { 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "data": { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG { 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "instance": "10.142.0.11:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "instance": "10.142.0.51:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG "0.0003999999999981204" 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG { 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "data": { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG { 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG "0.03373716962817191" 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906095" 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG { 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "data": { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG { 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "pod": "ds-cts-0" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "pod": "ds-idrepo-0" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG "0.1478462022886122" 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG { 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "data": { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG { 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "instance": "10.142.0.11:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "instance": "10.142.0.51:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG "1220116.3666666662" 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906095" 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG { 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "data": { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG { 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "instance": "10.142.0.11:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "instance": "10.142.0.51:9100" 12:08:15 DEBUG }, 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG "25804.8" 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906095" 12:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906095" 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 INFO [http_cmd]: http status code OK 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG --- status code --- 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG --- http response --- 12:08:15 DEBUG { 12:08:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG ----- output ----- 12:08:15 DEBUG "data": { 12:08:15 DEBUG { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "status": "success", 12:08:15 DEBUG "result": [ 12:08:15 DEBUG "data": { 12:08:15 DEBUG { 12:08:15 DEBUG "resultType": "vector", 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "result": [ 12:08:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:08:15 DEBUG { 12:08:15 DEBUG }, 12:08:15 DEBUG "metric": { 12:08:15 DEBUG "value": [ 12:08:15 DEBUG "pod": "ds-idrepo-0" 12:08:15 DEBUG 12:08:15 DEBUG }, 12:08:15 DEBUG 12:08:15 DEBUG "value": [ 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG [...] 12:08:15 DEBUG 12:08:15 DEBUG 12:08:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:08:15 DEBUG 12:08:15 DEBUG }, 12:08:15 DEBUG 12:08:15 DEBUG "value": [ 12:08:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG }, 12:08:15 DEBUG "14991777792" 12:08:15 DEBUG "value": [ 12:08:15 DEBUG ] 12:08:15 DEBUG 1735906095, 12:08:15 DEBUG } 12:08:15 DEBUG "758550528" 12:08:15 DEBUG ] 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:15 DEBUG ] 12:08:15 DEBUG } 12:08:15 DEBUG } 12:08:19 INFO 12:08:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:08:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:08:19 INFO [loop_until]: OK (rc = 0) 12:08:19 DEBUG --- stdout --- 12:08:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 173m 1% 3843Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 140m 0% 19061Mi 67% 12:08:19 DEBUG --- stderr --- 12:08:19 DEBUG 12:08:23 INFO 12:08:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:08:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:08:23 INFO [loop_until]: OK (rc = 0) 12:08:23 DEBUG --- stdout --- 12:08:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 76m 13479Mi rcs-645c86fdd6-j4rsk 98m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:08:23 DEBUG --- stderr --- 12:08:23 DEBUG 12:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906153" 12:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906153" 12:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906153" 12:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906153" 12:09:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906153" 12:09:13 ERROR [http_cmd]: ERROR 12:09:13 ERROR --- status code --- 12:09:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:09:13 ERROR --- http response --- 12:09:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:09:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:09:13 INFO [http_cmd]: http status code OK 12:09:13 DEBUG --- status code --- 12:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:13 DEBUG --- http response --- 12:09:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:13 INFO [http_cmd]: http status code OK 12:09:13 INFO [http_cmd]: http status code OK 12:09:13 DEBUG --- status code --- 12:09:13 DEBUG --- status code --- 12:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:13 DEBUG --- http response --- 12:09:13 DEBUG --- http response --- 12:09:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:13 INFO [http_cmd]: http status code OK 12:09:13 DEBUG --- status code --- 12:09:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:13 DEBUG --- http response --- 12:09:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- http response --- 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906154" 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:14 INFO [http_cmd]: http status code OK 12:09:14 DEBUG --- status code --- 12:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:14 DEBUG --- http response --- 12:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG "data": { 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "pod": "ds-cts-0" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG }, 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG "value": [ 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "436.31410710730637" 12:09:15 DEBUG ] 12:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906155" 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "pod": "ds-cts-0" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "0" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "instance": "10.142.0.11:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG "instance": "10.142.0.51:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "1788934.666666667" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906155" 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "instance": "10.142.0.11:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG "instance": "10.142.0.51:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "0" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "0.10521272281300235" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "instance": "10.142.0.11:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG "instance": "10.142.0.51:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "0.00043333333333540246" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "pod": "ds-cts-0" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906155" 12:09:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "0" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "instance": "10.142.0.11:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG "instance": "10.142.0.51:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "1772728.0333333332" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906155" 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "instance": "10.142.0.11:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG "instance": "10.142.0.51:9100" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "26760.533333333333" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906155" 12:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906155" 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "pod": "ds-idrepo-0" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "758550528" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:15 INFO [http_cmd]: http status code OK 12:09:15 DEBUG --- status code --- 12:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:09:15 DEBUG --- http response --- 12:09:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:09:15 DEBUG ----- output ----- 12:09:15 DEBUG { 12:09:15 DEBUG "status": "success", 12:09:15 DEBUG "data": { 12:09:15 DEBUG "resultType": "vector", 12:09:15 DEBUG "result": [ 12:09:15 DEBUG { 12:09:15 DEBUG "metric": { 12:09:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG [...] 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG 12:09:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:09:15 DEBUG }, 12:09:15 DEBUG "value": [ 12:09:15 DEBUG 1735906155, 12:09:15 DEBUG "14991679488" 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG ] 12:09:15 DEBUG } 12:09:15 DEBUG } 12:09:19 INFO 12:09:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:09:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:09:19 INFO [loop_until]: OK (rc = 0) 12:09:19 DEBUG --- stdout --- 12:09:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 164m 1% 3846Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 132m 0% 19061Mi 67% 12:09:19 DEBUG --- stderr --- 12:09:19 DEBUG 12:09:23 INFO 12:09:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:09:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:09:23 INFO [loop_until]: OK (rc = 0) 12:09:23 DEBUG --- stdout --- 12:09:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 259m 13478Mi rcs-645c86fdd6-j4rsk 101m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:09:23 DEBUG --- stderr --- 12:09:23 DEBUG 12:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906213" 12:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906213" 12:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906213" 12:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906213" 12:10:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906213" 12:10:13 INFO [http_cmd]: http status code OK 12:10:13 DEBUG --- status code --- 12:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:13 DEBUG --- http response --- 12:10:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:13 ERROR [http_cmd]: ERROR 12:10:13 ERROR --- status code --- 12:10:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:10:13 ERROR --- http response --- 12:10:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:10:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:10:13 INFO [http_cmd]: http status code OK 12:10:13 DEBUG --- status code --- 12:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:13 DEBUG --- http response --- 12:10:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:13 INFO [http_cmd]: http status code OK 12:10:13 DEBUG --- status code --- 12:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:13 DEBUG --- http response --- 12:10:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:13 INFO [http_cmd]: http status code OK 12:10:13 DEBUG --- status code --- 12:10:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:13 DEBUG --- http response --- 12:10:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 DEBUG --- http response --- 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906214" 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:14 INFO [http_cmd]: http status code OK 12:10:14 DEBUG --- status code --- 12:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:14 DEBUG --- http response --- 12:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- http response --- 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "556.3327674023769" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906215" 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "0" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "instance": "10.142.0.11:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "instance": "10.142.0.51:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "2384696.766666666" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906215" 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "instance": "10.142.0.11:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "instance": "10.142.0.51:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "0" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906215" 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "0.06387648921322081" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "instance": "10.142.0.11:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "instance": "10.142.0.51:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "0.00046666666666700016" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906215" 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "pod": "ds-cts-0" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "0" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "instance": "10.142.0.11:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "instance": "10.142.0.51:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "2367275.433333334" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906215" 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "instance": "10.142.0.11:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "instance": "10.142.0.51:9100" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "28262.4" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906215" 12:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906215" 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "pod": "ds-idrepo-0" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "758550528" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:15 INFO [http_cmd]: http status code OK 12:10:15 DEBUG --- status code --- 12:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:10:15 DEBUG --- http response --- 12:10:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:10:15 DEBUG ----- output ----- 12:10:15 DEBUG { 12:10:15 DEBUG "status": "success", 12:10:15 DEBUG "data": { 12:10:15 DEBUG "resultType": "vector", 12:10:15 DEBUG "result": [ 12:10:15 DEBUG { 12:10:15 DEBUG "metric": { 12:10:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG [...] 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG 12:10:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:10:15 DEBUG }, 12:10:15 DEBUG "value": [ 12:10:15 DEBUG 1735906215, 12:10:15 DEBUG "14991241216" 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG ] 12:10:15 DEBUG } 12:10:15 DEBUG } 12:10:19 INFO 12:10:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:10:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:10:19 INFO [loop_until]: OK (rc = 0) 12:10:19 DEBUG --- stdout --- 12:10:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 209m 1% 3863Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 170m 0% 19055Mi 67% 12:10:19 DEBUG --- stderr --- 12:10:19 DEBUG 12:10:23 INFO 12:10:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:10:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:10:23 INFO [loop_until]: OK (rc = 0) 12:10:23 DEBUG --- stdout --- 12:10:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 116m 13478Mi rcs-645c86fdd6-j4rsk 108m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:10:23 DEBUG --- stderr --- 12:10:23 DEBUG 12:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906273" 12:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906273" 12:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906273" 12:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906273" 12:11:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906273" 12:11:13 INFO [http_cmd]: http status code OK 12:11:13 DEBUG --- status code --- 12:11:13 INFO [http_cmd]: http status code OK 12:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:13 DEBUG --- status code --- 12:11:13 ERROR [http_cmd]: ERROR 12:11:13 DEBUG --- http response --- 12:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:13 ERROR --- status code --- 12:11:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:13 INFO [http_cmd]: http status code OK 12:11:13 INFO [http_cmd]: http status code OK 12:11:13 DEBUG --- http response --- 12:11:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:11:13 DEBUG --- status code --- 12:11:13 DEBUG --- status code --- 12:11:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:13 ERROR --- http response --- 12:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:11:13 DEBUG --- http response --- 12:11:13 DEBUG --- http response --- 12:11:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:11:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906274" 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:14 INFO [http_cmd]: http status code OK 12:11:14 DEBUG --- status code --- 12:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:14 DEBUG --- http response --- 12:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "data": { 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG [...] 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "pod": "ds-cts-0" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG "422.093924799324" 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906275" 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "data": { 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG [...] 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "pod": "ds-cts-0" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG "0" 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "data": { 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "instance": "10.142.0.11:9100" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG [...] 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "instance": "10.142.0.51:9100" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG "2301373.366666666" 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906275" 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "data": { 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "instance": "10.142.0.11:9100" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG [...] 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "instance": "10.142.0.51:9100" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG "0.001000000000001459" 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906275" 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG { 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG "data": { 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "data": { 12:11:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG }, 12:11:15 DEBUG "result": [ 12:11:15 DEBUG "value": [ 12:11:15 DEBUG { 12:11:15 DEBUG 12:11:15 DEBUG "metric": { 12:11:15 DEBUG 12:11:15 DEBUG "instance": "10.142.0.11:9100" 12:11:15 DEBUG 12:11:15 DEBUG }, 12:11:15 DEBUG [...] 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:11:15 DEBUG [...] 12:11:15 DEBUG }, 12:11:15 DEBUG 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG 12:11:15 DEBUG "0.09241445637646872" 12:11:15 DEBUG "instance": "10.142.0.51:9100" 12:11:15 DEBUG ] 12:11:15 DEBUG }, 12:11:15 DEBUG } 12:11:15 DEBUG "value": [ 12:11:15 DEBUG ] 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG } 12:11:15 DEBUG "0.00046666666666700016" 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906275" 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "data": { 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG [...] 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "pod": "ds-cts-0" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG "0.0016290702661529574" 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "data": { 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "instance": "10.142.0.11:9100" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG [...] 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "instance": "10.142.0.51:9100" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG "2284507.5333333327" 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906275" 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "data": { 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "instance": "10.142.0.11:9100" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG [...] 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "instance": "10.142.0.51:9100" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG "28808.533333333333" 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906275" 12:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906275" 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "data": { 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "pod": "ds-idrepo-0" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG [...] 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG "758550528" 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:15 INFO [http_cmd]: http status code OK 12:11:15 DEBUG --- status code --- 12:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:11:15 DEBUG --- http response --- 12:11:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:11:15 DEBUG ----- output ----- 12:11:15 DEBUG { 12:11:15 DEBUG "status": "success", 12:11:15 DEBUG "data": { 12:11:15 DEBUG "resultType": "vector", 12:11:15 DEBUG "result": [ 12:11:15 DEBUG { 12:11:15 DEBUG "metric": { 12:11:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG [...] 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG 12:11:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:11:15 DEBUG }, 12:11:15 DEBUG "value": [ 12:11:15 DEBUG 1735906275, 12:11:15 DEBUG "14990458880" 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG ] 12:11:15 DEBUG } 12:11:15 DEBUG } 12:11:19 INFO 12:11:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:11:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:11:19 INFO [loop_until]: OK (rc = 0) 12:11:19 DEBUG --- stdout --- 12:11:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 218m 1% 3866Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 195m 0% 19058Mi 67% 12:11:19 DEBUG --- stderr --- 12:11:19 DEBUG 12:11:23 INFO 12:11:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:11:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:11:23 INFO [loop_until]: OK (rc = 0) 12:11:23 DEBUG --- stdout --- 12:11:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 108m 13478Mi rcs-645c86fdd6-j4rsk 99m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:11:23 DEBUG --- stderr --- 12:11:23 DEBUG 12:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906333" 12:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906333" 12:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906333" 12:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906333" 12:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906333" 12:12:13 INFO [http_cmd]: http status code OK 12:12:13 DEBUG --- status code --- 12:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:13 DEBUG --- http response --- 12:12:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:13 INFO [http_cmd]: http status code OK 12:12:13 INFO [http_cmd]: http status code OK 12:12:13 DEBUG --- status code --- 12:12:13 DEBUG --- status code --- 12:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:13 DEBUG --- http response --- 12:12:13 DEBUG --- http response --- 12:12:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:13 INFO [http_cmd]: http status code OK 12:12:13 ERROR [http_cmd]: ERROR 12:12:13 DEBUG --- status code --- 12:12:13 ERROR --- status code --- 12:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:12:13 DEBUG --- http response --- 12:12:13 ERROR --- http response --- 12:12:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:12:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906334" 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:14 INFO [http_cmd]: http status code OK 12:12:14 DEBUG --- status code --- 12:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:14 DEBUG --- http response --- 12:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "pod": "ds-cts-0" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "638.8022457891453" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906335" 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "pod": "ds-cts-0" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "0" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "instance": "10.142.0.11:9100" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "instance": "10.142.0.51:9100" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "1343776.9333333338" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906335" 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "instance": "10.142.0.11:9100" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "instance": "10.142.0.51:9100" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "0.0009999999999995642" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906335" 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "instance": "10.142.0.11:9100" 12:12:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:12:15 DEBUG }, 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "instance": "10.142.0.51:9100" 12:12:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:12:15 DEBUG }, 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "0.1425588816386054" 12:12:15 DEBUG "0.0012666666666670305" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906335" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "pod": "ds-cts-0" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "0.0027336868371790335" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906335" 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "instance": "10.142.0.11:9100" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "instance": "10.142.0.51:9100" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "1329239.8999999994" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906335" 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "instance": "10.142.0.11:9100" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "instance": "10.142.0.51:9100" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "40277.33333333333" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906335" 12:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906335" 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "14992789504" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:15 INFO [http_cmd]: http status code OK 12:12:15 DEBUG --- status code --- 12:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:12:15 DEBUG --- http response --- 12:12:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:12:15 DEBUG ----- output ----- 12:12:15 DEBUG { 12:12:15 DEBUG "status": "success", 12:12:15 DEBUG "data": { 12:12:15 DEBUG "resultType": "vector", 12:12:15 DEBUG "result": [ 12:12:15 DEBUG { 12:12:15 DEBUG "metric": { 12:12:15 DEBUG "pod": "ds-idrepo-0" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG [...] 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG 12:12:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:12:15 DEBUG }, 12:12:15 DEBUG "value": [ 12:12:15 DEBUG 1735906335, 12:12:15 DEBUG "758550528" 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG ] 12:12:15 DEBUG } 12:12:15 DEBUG } 12:12:19 INFO 12:12:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:12:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:12:19 INFO [loop_until]: OK (rc = 0) 12:12:19 DEBUG --- stdout --- 12:12:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 220m 1% 3886Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 227m 0% 19055Mi 67% 12:12:19 DEBUG --- stderr --- 12:12:19 DEBUG 12:12:23 INFO 12:12:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:12:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:12:23 INFO [loop_until]: OK (rc = 0) 12:12:23 DEBUG --- stdout --- 12:12:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 168m 13478Mi rcs-645c86fdd6-j4rsk 126m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:12:23 DEBUG --- stderr --- 12:12:23 DEBUG 12:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906393" 12:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906393" 12:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906393" 12:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906393" 12:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906393" 12:13:13 INFO [http_cmd]: http status code OK 12:13:13 DEBUG --- status code --- 12:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:13 INFO [http_cmd]: http status code OK 12:13:13 DEBUG --- http response --- 12:13:13 DEBUG --- status code --- 12:13:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:13 DEBUG --- http response --- 12:13:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:13 INFO [http_cmd]: http status code OK 12:13:13 DEBUG --- status code --- 12:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:13 DEBUG --- http response --- 12:13:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:13 ERROR [http_cmd]: ERROR 12:13:13 ERROR --- status code --- 12:13:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:13:13 ERROR --- http response --- 12:13:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:13:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:13:13 INFO [http_cmd]: http status code OK 12:13:13 DEBUG --- status code --- 12:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:13 DEBUG --- http response --- 12:13:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906394" 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 INFO [http_cmd]: http status code OK 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG --- status code --- 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG --- http response --- 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 DEBUG --- status code --- 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "pod": "ds-cts-0" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "pod": "ds-idrepo-0" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "1290.03359462486" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906395" 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "pod": "ds-cts-0" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "pod": "ds-idrepo-0" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "0" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "instance": "10.142.0.11:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "instance": "10.142.0.51:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "1239306.8333333333" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906395" 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "instance": "10.142.0.11:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "instance": "10.142.0.51:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "0" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "instance": "10.142.0.11:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "instance": "10.142.0.51:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "0.00040000000000190995" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "0.19811651525178658" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "pod": "ds-cts-0" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "pod": "ds-idrepo-0" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "0.005301931691276807" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906395" 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "instance": "10.142.0.11:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "instance": "10.142.0.51:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "1222354.2" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906395" 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "instance": "10.142.0.11:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "instance": "10.142.0.51:9100" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "23620.266666666666" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906395" 12:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906395" 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "14991663104" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:15 INFO [http_cmd]: http status code OK 12:13:15 DEBUG --- status code --- 12:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:13:15 DEBUG --- http response --- 12:13:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:13:15 DEBUG ----- output ----- 12:13:15 DEBUG { 12:13:15 DEBUG "status": "success", 12:13:15 DEBUG "data": { 12:13:15 DEBUG "resultType": "vector", 12:13:15 DEBUG "result": [ 12:13:15 DEBUG { 12:13:15 DEBUG "metric": { 12:13:15 DEBUG "pod": "ds-idrepo-0" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG [...] 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG 12:13:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:13:15 DEBUG }, 12:13:15 DEBUG "value": [ 12:13:15 DEBUG 1735906395, 12:13:15 DEBUG "758550528" 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG ] 12:13:15 DEBUG } 12:13:15 DEBUG } 12:13:19 INFO 12:13:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:13:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:13:19 INFO [loop_until]: OK (rc = 0) 12:13:19 DEBUG --- stdout --- 12:13:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 188m 1% 3827Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 159m 0% 19055Mi 67% 12:13:19 DEBUG --- stderr --- 12:13:19 DEBUG 12:13:23 INFO 12:13:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:13:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:13:23 INFO [loop_until]: OK (rc = 0) 12:13:23 DEBUG --- stdout --- 12:13:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 93m 13478Mi rcs-645c86fdd6-j4rsk 103m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:13:23 DEBUG --- stderr --- 12:13:23 DEBUG 12:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906453" 12:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906453" 12:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906453" 12:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906453" 12:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906453" 12:14:13 INFO [http_cmd]: http status code OK 12:14:13 DEBUG --- status code --- 12:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:13 DEBUG --- http response --- 12:14:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:13 INFO [http_cmd]: http status code OK 12:14:13 INFO [http_cmd]: http status code OK 12:14:13 ERROR [http_cmd]: ERROR 12:14:13 DEBUG --- status code --- 12:14:13 DEBUG --- status code --- 12:14:13 ERROR --- status code --- 12:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:14:13 DEBUG --- http response --- 12:14:13 DEBUG --- http response --- 12:14:13 ERROR --- http response --- 12:14:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:14:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:14:13 INFO [http_cmd]: http status code OK 12:14:13 DEBUG --- status code --- 12:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:13 DEBUG --- http response --- 12:14:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: http status code OK 12:14:14 DEBUG --- status code --- 12:14:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:14 DEBUG --- http response --- 12:14:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: http status code OK 12:14:14 DEBUG --- status code --- 12:14:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:14 DEBUG --- http response --- 12:14:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: http status code OK 12:14:14 DEBUG --- status code --- 12:14:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:14 DEBUG --- http response --- 12:14:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: http status code OK 12:14:14 DEBUG --- status code --- 12:14:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:14 DEBUG --- http response --- 12:14:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: http status code OK 12:14:14 DEBUG --- status code --- 12:14:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:14 DEBUG --- http response --- 12:14:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: http status code OK 12:14:14 DEBUG --- status code --- 12:14:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:14 DEBUG --- http response --- 12:14:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: http status code OK 12:14:14 DEBUG --- status code --- 12:14:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:14 DEBUG --- http response --- 12:14:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: http status code OK 12:14:14 DEBUG --- status code --- 12:14:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:14 DEBUG --- http response --- 12:14:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: http status code OK 12:14:14 DEBUG --- status code --- 12:14:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:14 DEBUG --- http response --- 12:14:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906454" 12:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906454" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG { 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "data": { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG { 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "pod": "ds-idrepo-0" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG [...] 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 DEBUG 12:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:14:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "555.9650344925731" 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906455" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG { 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "data": { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG { 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "pod": "ds-idrepo-0" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "0" 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG { 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "data": { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG { 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "instance": "10.142.0.11:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG "instance": "10.142.0.51:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "571463.0333333333" 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906455" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG { 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "data": { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG { 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "instance": "10.142.0.11:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG "instance": "10.142.0.51:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "0" 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG { 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "data": { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG { 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "instance": "10.142.0.11:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG "instance": "10.142.0.51:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "0.00043333333333066547" 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG { 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "data": { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG { 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "0.1242043277015459" 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG { 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "data": { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG { 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "pod": "ds-idrepo-0" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "0" 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906455" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG { 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "data": { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG { 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "instance": "10.142.0.11:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG "instance": "10.142.0.51:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "554064.9" 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906455" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG { 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "data": { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG { 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "instance": "10.142.0.11:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG "instance": "10.142.0.51:9100" 12:14:15 DEBUG }, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "18158.93333333333" 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906455" 12:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906455" 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 INFO [http_cmd]: http status code OK 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG --- status code --- 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG --- http response --- 12:14:15 DEBUG { 12:14:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG ----- output ----- 12:14:15 DEBUG "data": { 12:14:15 DEBUG { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "status": "success", 12:14:15 DEBUG "result": [ 12:14:15 DEBUG "data": { 12:14:15 DEBUG { 12:14:15 DEBUG "resultType": "vector", 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "result": [ 12:14:15 DEBUG "pod": "ds-idrepo-0" 12:14:15 DEBUG { 12:14:15 DEBUG }, 12:14:15 DEBUG "metric": { 12:14:15 DEBUG "value": [ 12:14:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:14:15 DEBUG 12:14:15 DEBUG }, 12:14:15 DEBUG 12:14:15 DEBUG "value": [ 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG [...] 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG 12:14:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:14:15 DEBUG 12:14:15 DEBUG }, 12:14:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:14:15 DEBUG "value": [ 12:14:15 DEBUG }, 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG "value": [ 12:14:15 DEBUG "758550528" 12:14:15 DEBUG 1735906455, 12:14:15 DEBUG ] 12:14:15 DEBUG "14991011840" 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 DEBUG ] 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:15 DEBUG } 12:14:19 INFO 12:14:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:14:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:14:19 INFO [loop_until]: OK (rc = 0) 12:14:19 DEBUG --- stdout --- 12:14:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 161m 1% 3848Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 516m 1% 19054Mi 66% 12:14:19 DEBUG --- stderr --- 12:14:19 DEBUG 12:14:23 INFO 12:14:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:14:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:14:23 INFO [loop_until]: OK (rc = 0) 12:14:23 DEBUG --- stdout --- 12:14:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:14:23 DEBUG --- stderr --- 12:14:23 DEBUG 12:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906513" 12:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906513" 12:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906513" 12:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906513" 12:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906513" 12:15:13 INFO [http_cmd]: http status code OK 12:15:13 DEBUG --- status code --- 12:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:13 DEBUG --- http response --- 12:15:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:13 INFO [http_cmd]: http status code OK 12:15:13 DEBUG --- status code --- 12:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:13 DEBUG --- http response --- 12:15:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:13 INFO [http_cmd]: http status code OK 12:15:13 DEBUG --- status code --- 12:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:13 DEBUG --- http response --- 12:15:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:13 ERROR [http_cmd]: ERROR 12:15:13 INFO [http_cmd]: http status code OK 12:15:13 ERROR --- status code --- 12:15:13 DEBUG --- status code --- 12:15:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:13 ERROR --- http response --- 12:15:13 DEBUG --- http response --- 12:15:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:15:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:15:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906514" 12:15:14 INFO [http_cmd]: http status code OK 12:15:14 DEBUG --- status code --- 12:15:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:14 DEBUG --- http response --- 12:15:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906514" 12:15:14 INFO [http_cmd]: http status code OK 12:15:14 DEBUG --- status code --- 12:15:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:14 DEBUG --- http response --- 12:15:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906514" 12:15:14 INFO [http_cmd]: http status code OK 12:15:14 DEBUG --- status code --- 12:15:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:14 DEBUG --- http response --- 12:15:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906514" 12:15:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906514" 12:15:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906514" 12:15:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906514" 12:15:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906514" 12:15:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906514" 12:15:14 INFO [http_cmd]: http status code OK 12:15:14 DEBUG --- status code --- 12:15:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:14 DEBUG --- http response --- 12:15:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:14 INFO [http_cmd]: http status code OK 12:15:14 INFO [http_cmd]: http status code OK 12:15:14 DEBUG --- status code --- 12:15:14 DEBUG --- status code --- 12:15:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:14 DEBUG --- http response --- 12:15:14 DEBUG --- http response --- 12:15:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:14 INFO [http_cmd]: http status code OK 12:15:14 DEBUG --- status code --- 12:15:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:14 DEBUG --- http response --- 12:15:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:14 INFO [http_cmd]: http status code OK 12:15:14 DEBUG --- status code --- 12:15:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:14 DEBUG --- http response --- 12:15:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:14 INFO [http_cmd]: http status code OK 12:15:14 DEBUG --- status code --- 12:15:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:14 DEBUG --- http response --- 12:15:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- status code --- 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- http response --- 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG --- http response --- 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "pod": "ds-cts-0" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "313.6884395011797" 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "pod": "ds-cts-0" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "0" 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "instance": "10.142.0.11:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "instance": "10.142.0.51:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "92783.33333333333" 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "instance": "10.142.0.11:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "instance": "10.142.0.51:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "0.00033333333333397756" 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:15:15 DEBUG "instance": "10.142.0.11:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:15:15 DEBUG "instance": "10.142.0.51:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "0.5608193629444596" 12:15:15 DEBUG "0.00043333333333161285" 12:15:15 DEBUG ] 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "pod": "ds-cts-0" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "0.0017425688939417718" 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "instance": "10.142.0.11:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "instance": "10.142.0.51:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "76983.70000000001" 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "instance": "10.142.0.11:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "instance": "10.142.0.51:9100" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "26624" 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906515" 12:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906515" 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "pod": "ds-idrepo-0" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "758550528" 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:15 INFO [http_cmd]: http status code OK 12:15:15 DEBUG --- status code --- 12:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:15:15 DEBUG --- http response --- 12:15:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:15:15 DEBUG ----- output ----- 12:15:15 DEBUG { 12:15:15 DEBUG "status": "success", 12:15:15 DEBUG "data": { 12:15:15 DEBUG "resultType": "vector", 12:15:15 DEBUG "result": [ 12:15:15 DEBUG { 12:15:15 DEBUG "metric": { 12:15:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG [...] 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG 12:15:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:15:15 DEBUG }, 12:15:15 DEBUG "value": [ 12:15:15 DEBUG 1735906515, 12:15:15 DEBUG "14990569472" 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG ] 12:15:15 DEBUG } 12:15:15 DEBUG } 12:15:19 INFO 12:15:19 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:15:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:15:20 INFO [loop_until]: OK (rc = 0) 12:15:20 DEBUG --- stdout --- 12:15:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 145m 0% 3848Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 117m 0% 19053Mi 66% 12:15:20 DEBUG --- stderr --- 12:15:20 DEBUG 12:15:23 INFO 12:15:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:15:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:15:23 INFO [loop_until]: OK (rc = 0) 12:15:23 DEBUG --- stdout --- 12:15:23 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 26m 13479Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:15:23 DEBUG --- stderr --- 12:15:23 DEBUG 12:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906573" 12:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906573" 12:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906573" 12:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906573" 12:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906573" 12:16:13 INFO [http_cmd]: http status code OK 12:16:13 INFO [http_cmd]: http status code OK 12:16:13 INFO [http_cmd]: http status code OK 12:16:13 INFO [http_cmd]: http status code OK 12:16:13 ERROR [http_cmd]: ERROR 12:16:13 DEBUG --- status code --- 12:16:13 DEBUG --- status code --- 12:16:13 DEBUG --- status code --- 12:16:13 DEBUG --- status code --- 12:16:13 ERROR --- status code --- 12:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:13 DEBUG --- http response --- 12:16:13 DEBUG --- http response --- 12:16:13 DEBUG --- http response --- 12:16:13 DEBUG --- http response --- 12:16:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:16:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:13 ERROR --- http response --- 12:16:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:16:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:16:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906574" 12:16:14 INFO [http_cmd]: http status code OK 12:16:14 DEBUG --- status code --- 12:16:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:14 DEBUG --- http response --- 12:16:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906574" 12:16:14 INFO [http_cmd]: http status code OK 12:16:14 DEBUG --- status code --- 12:16:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:14 DEBUG --- http response --- 12:16:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906574" 12:16:14 INFO [http_cmd]: http status code OK 12:16:14 DEBUG --- status code --- 12:16:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:14 DEBUG --- http response --- 12:16:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906574" 12:16:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906574" 12:16:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906574" 12:16:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906574" 12:16:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906574" 12:16:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906574" 12:16:14 INFO [http_cmd]: http status code OK 12:16:14 DEBUG --- status code --- 12:16:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:14 DEBUG --- http response --- 12:16:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:14 INFO [http_cmd]: http status code OK 12:16:14 INFO [http_cmd]: http status code OK 12:16:14 INFO [http_cmd]: http status code OK 12:16:14 DEBUG --- status code --- 12:16:14 INFO [http_cmd]: http status code OK 12:16:14 DEBUG --- status code --- 12:16:14 DEBUG --- status code --- 12:16:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:14 DEBUG --- status code --- 12:16:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:14 DEBUG --- http response --- 12:16:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:14 DEBUG --- http response --- 12:16:14 INFO [http_cmd]: http status code OK 12:16:14 DEBUG --- http response --- 12:16:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:14 DEBUG --- http response --- 12:16:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:14 DEBUG --- status code --- 12:16:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:14 DEBUG --- http response --- 12:16:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG { 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "data": { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG { 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "pod": "ds-cts-0" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG "389.5140583890702" 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG { 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "data": { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG { 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "pod": "ds-cts-0" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG "0" 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG { 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "data": { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG { 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "instance": "10.142.0.11:9100" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "instance": "10.142.0.51:9100" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG "93233.93333333335" 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG { 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "data": { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG { 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "instance": "10.142.0.11:9100" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "instance": "10.142.0.51:9100" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG "0" 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG { 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG "data": { 12:16:15 DEBUG { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG "data": { 12:16:15 DEBUG { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "result": [ 12:16:15 DEBUG "instance": "10.142.0.11:9100" 12:16:15 DEBUG { 12:16:15 DEBUG }, 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "value": [ 12:16:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:16:15 DEBUG 12:16:15 DEBUG }, 12:16:15 DEBUG 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "instance": "10.142.0.51:9100" 12:16:15 DEBUG 12:16:15 DEBUG }, 12:16:15 DEBUG 12:16:15 DEBUG "value": [ 12:16:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG }, 12:16:15 DEBUG "0.0005666666666617933" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG "value": [ 12:16:15 DEBUG ] 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG } 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG "0.05943834271553895" 12:16:15 DEBUG ] 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG } 12:16:15 DEBUG { 12:16:15 DEBUG } 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "data": { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG { 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "pod": "ds-cts-0" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG "0.000588360224451541" 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG { 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "data": { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG { 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "instance": "10.142.0.11:9100" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "instance": "10.142.0.51:9100" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG "76530.63333333335" 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG { 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "data": { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG { 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "instance": "10.142.0.11:9100" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "instance": "10.142.0.51:9100" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG "25122.133333333335" 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906575" 12:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906575" 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG { 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "data": { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG { 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "pod": "ds-idrepo-0" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG "758550528" 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:15 INFO [http_cmd]: http status code OK 12:16:15 DEBUG --- status code --- 12:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:16:15 DEBUG --- http response --- 12:16:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:16:15 DEBUG ----- output ----- 12:16:15 DEBUG { 12:16:15 DEBUG "status": "success", 12:16:15 DEBUG "data": { 12:16:15 DEBUG "resultType": "vector", 12:16:15 DEBUG "result": [ 12:16:15 DEBUG { 12:16:15 DEBUG "metric": { 12:16:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG [...] 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG 12:16:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:16:15 DEBUG }, 12:16:15 DEBUG "value": [ 12:16:15 DEBUG 1735906575, 12:16:15 DEBUG "14991364096" 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG ] 12:16:15 DEBUG } 12:16:15 DEBUG } 12:16:20 INFO 12:16:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:16:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:16:20 INFO [loop_until]: OK (rc = 0) 12:16:20 DEBUG --- stdout --- 12:16:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 109m 0% 3851Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 74m 0% 19054Mi 66% 12:16:20 DEBUG --- stderr --- 12:16:20 DEBUG 12:16:23 INFO 12:16:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:16:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:16:24 INFO [loop_until]: OK (rc = 0) 12:16:24 DEBUG --- stdout --- 12:16:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13479Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:16:24 DEBUG --- stderr --- 12:16:24 DEBUG 12:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906633" 12:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906633" 12:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906633" 12:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906633" 12:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906633" 12:17:13 INFO [http_cmd]: http status code OK 12:17:13 DEBUG --- status code --- 12:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:13 DEBUG --- http response --- 12:17:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:13 INFO [http_cmd]: http status code OK 12:17:13 DEBUG --- status code --- 12:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:13 DEBUG --- http response --- 12:17:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:13 INFO [http_cmd]: http status code OK 12:17:13 DEBUG --- status code --- 12:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:13 DEBUG --- http response --- 12:17:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:13 INFO [http_cmd]: http status code OK 12:17:13 DEBUG --- status code --- 12:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:13 DEBUG --- http response --- 12:17:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:13 ERROR [http_cmd]: ERROR 12:17:13 ERROR --- status code --- 12:17:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:17:13 ERROR --- http response --- 12:17:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:17:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:17:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906634" 12:17:14 INFO [http_cmd]: http status code OK 12:17:14 DEBUG --- status code --- 12:17:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:14 DEBUG --- http response --- 12:17:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906634" 12:17:14 INFO [http_cmd]: http status code OK 12:17:14 DEBUG --- status code --- 12:17:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:14 DEBUG --- http response --- 12:17:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906634" 12:17:14 INFO [http_cmd]: http status code OK 12:17:14 DEBUG --- status code --- 12:17:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:14 DEBUG --- http response --- 12:17:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906634" 12:17:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906634" 12:17:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906634" 12:17:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906634" 12:17:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906634" 12:17:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906634" 12:17:14 INFO [http_cmd]: http status code OK 12:17:14 DEBUG --- status code --- 12:17:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:14 DEBUG --- http response --- 12:17:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:14 INFO [http_cmd]: http status code OK 12:17:14 DEBUG --- status code --- 12:17:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:14 DEBUG --- http response --- 12:17:14 INFO [http_cmd]: http status code OK 12:17:14 DEBUG --- status code --- 12:17:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:14 INFO [http_cmd]: http status code OK 12:17:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:14 DEBUG --- http response --- 12:17:14 DEBUG --- status code --- 12:17:14 INFO [http_cmd]: http status code OK 12:17:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:14 INFO [http_cmd]: http status code OK 12:17:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:14 DEBUG --- status code --- 12:17:14 DEBUG --- status code --- 12:17:14 DEBUG --- http response --- 12:17:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:14 DEBUG --- http response --- 12:17:14 DEBUG --- http response --- 12:17:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG --- http response --- 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG --- http response --- 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG { 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG "data": { 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG "result": [ 12:17:15 DEBUG { 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "pod": "ds-cts-0" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG "421.4653603569915" 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906635" 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG { 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG "data": { 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG "result": [ 12:17:15 DEBUG { 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "pod": "ds-cts-0" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG "0" 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG { 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG "data": { 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG "result": [ 12:17:15 DEBUG { 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "instance": "10.142.0.11:9100" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG "instance": "10.142.0.51:9100" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG "95927.73333333335" 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906635" 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG { 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG "data": { 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG "result": [ 12:17:15 DEBUG { 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "instance": "10.142.0.11:9100" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG "instance": "10.142.0.51:9100" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG "0.001666666666666572" 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG { 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG "data": { 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG { 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG "result": [ 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG { 12:17:15 DEBUG "data": { 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG "pod": "ds-cts-0" 12:17:15 DEBUG "result": [ 12:17:15 DEBUG { 12:17:15 DEBUG }, 12:17:15 DEBUG { 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG "value": [ 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "data": { 12:17:15 DEBUG 12:17:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG 12:17:15 DEBUG }, 12:17:15 DEBUG "result": [ 12:17:15 DEBUG 12:17:15 DEBUG "value": [ 12:17:15 DEBUG { 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG "metric": { 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG "instance": "10.142.0.11:9100" 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG }, 12:17:15 DEBUG 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906635" 12:17:15 DEBUG "value": [ 12:17:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:17:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:17:15 DEBUG 12:17:15 DEBUG }, 12:17:15 DEBUG }, 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG "value": [ 12:17:15 DEBUG "value": [ 12:17:15 DEBUG [...] 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG 12:17:15 DEBUG "0.029489231745022156" 12:17:15 DEBUG ] 12:17:15 DEBUG 12:17:15 DEBUG } 12:17:15 DEBUG "0" 12:17:15 DEBUG 12:17:15 DEBUG ] 12:17:15 DEBUG ] 12:17:15 DEBUG "instance": "10.142.0.51:9100" 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 DEBUG }, 12:17:15 DEBUG } 12:17:15 DEBUG ] 12:17:15 DEBUG "value": [ 12:17:15 DEBUG } 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG } 12:17:15 DEBUG "0.0013333333333406473" 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG { 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG "data": { 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG "result": [ 12:17:15 DEBUG { 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "instance": "10.142.0.11:9100" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG "instance": "10.142.0.51:9100" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG "80678.76666666669" 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906635" 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG { 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG "data": { 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG "result": [ 12:17:15 DEBUG { 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "instance": "10.142.0.11:9100" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG "instance": "10.142.0.51:9100" 12:17:15 DEBUG }, 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG "37956.26666666666" 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906635" 12:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906635" 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 INFO [http_cmd]: http status code OK 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG --- status code --- 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG --- http response --- 12:17:15 DEBUG { 12:17:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG ----- output ----- 12:17:15 DEBUG "data": { 12:17:15 DEBUG { 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG "status": "success", 12:17:15 DEBUG "result": [ 12:17:15 DEBUG "data": { 12:17:15 DEBUG { 12:17:15 DEBUG "resultType": "vector", 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "result": [ 12:17:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:17:15 DEBUG { 12:17:15 DEBUG }, 12:17:15 DEBUG "metric": { 12:17:15 DEBUG "value": [ 12:17:15 DEBUG "pod": "ds-idrepo-0" 12:17:15 DEBUG 12:17:15 DEBUG }, 12:17:15 DEBUG 12:17:15 DEBUG "value": [ 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG [...] 12:17:15 DEBUG 12:17:15 DEBUG 12:17:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:17:15 DEBUG 12:17:15 DEBUG }, 12:17:15 DEBUG 12:17:15 DEBUG "value": [ 12:17:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG }, 12:17:15 DEBUG "14990209024" 12:17:15 DEBUG "value": [ 12:17:15 DEBUG ] 12:17:15 DEBUG 1735906635, 12:17:15 DEBUG } 12:17:15 DEBUG "758550528" 12:17:15 DEBUG ] 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:15 DEBUG ] 12:17:15 DEBUG } 12:17:15 DEBUG } 12:17:20 INFO 12:17:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:17:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:17:20 INFO [loop_until]: OK (rc = 0) 12:17:20 DEBUG --- stdout --- 12:17:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 107m 0% 3854Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 79m 0% 19052Mi 66% 12:17:20 DEBUG --- stderr --- 12:17:20 DEBUG 12:17:24 INFO 12:17:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:17:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:17:24 INFO [loop_until]: OK (rc = 0) 12:17:24 DEBUG --- stdout --- 12:17:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13479Mi rcs-645c86fdd6-j4rsk 21m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:17:24 DEBUG --- stderr --- 12:17:24 DEBUG 12:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906693" 12:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906693" 12:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906693" 12:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906693" 12:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906693" 12:18:13 INFO [http_cmd]: http status code OK 12:18:13 DEBUG --- status code --- 12:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:13 DEBUG --- http response --- 12:18:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:13 ERROR [http_cmd]: ERROR 12:18:13 INFO [http_cmd]: http status code OK 12:18:13 INFO [http_cmd]: http status code OK 12:18:13 ERROR --- status code --- 12:18:13 INFO [http_cmd]: http status code OK 12:18:13 DEBUG --- status code --- 12:18:13 DEBUG --- status code --- 12:18:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:18:13 DEBUG --- status code --- 12:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:13 ERROR --- http response --- 12:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:13 DEBUG --- http response --- 12:18:13 DEBUG --- http response --- 12:18:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:18:13 DEBUG --- http response --- 12:18:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:18:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906694" 12:18:14 INFO [http_cmd]: http status code OK 12:18:14 DEBUG --- status code --- 12:18:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:14 DEBUG --- http response --- 12:18:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906694" 12:18:14 INFO [http_cmd]: http status code OK 12:18:14 DEBUG --- status code --- 12:18:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:14 DEBUG --- http response --- 12:18:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906694" 12:18:14 INFO [http_cmd]: http status code OK 12:18:14 DEBUG --- status code --- 12:18:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:14 DEBUG --- http response --- 12:18:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906694" 12:18:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906694" 12:18:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906694" 12:18:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906694" 12:18:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906694" 12:18:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906694" 12:18:14 INFO [http_cmd]: http status code OK 12:18:14 DEBUG --- status code --- 12:18:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:14 DEBUG --- http response --- 12:18:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:14 INFO [http_cmd]: http status code OK 12:18:14 DEBUG --- status code --- 12:18:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:14 DEBUG --- http response --- 12:18:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:14 INFO [http_cmd]: http status code OK 12:18:14 DEBUG --- status code --- 12:18:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:14 DEBUG --- http response --- 12:18:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:14 INFO [http_cmd]: http status code OK 12:18:14 DEBUG --- status code --- 12:18:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:14 DEBUG --- http response --- 12:18:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:14 INFO [http_cmd]: http status code OK 12:18:14 DEBUG --- status code --- 12:18:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:14 DEBUG --- http response --- 12:18:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:14 INFO [http_cmd]: http status code OK 12:18:14 DEBUG --- status code --- 12:18:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:14 DEBUG --- http response --- 12:18:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 DEBUG --- status code --- 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906695" 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG { 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG "data": { 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG "result": [ 12:18:15 DEBUG { 12:18:15 DEBUG "metric": { 12:18:15 DEBUG "pod": "ds-cts-0" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG [...] 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG "464.6756856405457" 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG { 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG "data": { 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG "result": [ 12:18:15 DEBUG { 12:18:15 DEBUG "metric": { 12:18:15 DEBUG "pod": "ds-cts-0" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG [...] 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG "0" 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG { 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG "data": { 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG "result": [ 12:18:15 DEBUG { 12:18:15 DEBUG "metric": { 12:18:15 DEBUG "instance": "10.142.0.11:9100" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG [...] 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG "instance": "10.142.0.51:9100" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG "850802.8333333335" 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906695" 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG { 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG "data": { 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG "result": [ 12:18:15 DEBUG { 12:18:15 DEBUG "metric": { 12:18:15 DEBUG "instance": "10.142.0.11:9100" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG [...] 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG "instance": "10.142.0.51:9100" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG "0.0003333333333330302" 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906695" 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906695" 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG { 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG "data": { 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG "result": [ 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG { 12:18:15 DEBUG { 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG "metric": { 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:18:15 DEBUG "instance": "10.142.0.11:9100" 12:18:15 DEBUG "data": { 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG }, 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG { 12:18:15 DEBUG "value": [ 12:18:15 DEBUG "result": [ 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG 12:18:15 DEBUG { 12:18:15 DEBUG "data": { 12:18:15 DEBUG 12:18:15 DEBUG "metric": { 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG 12:18:15 DEBUG "pod": "ds-cts-0" 12:18:15 DEBUG "result": [ 12:18:15 DEBUG [...] 12:18:15 DEBUG }, 12:18:15 DEBUG { 12:18:15 DEBUG 12:18:15 DEBUG "value": [ 12:18:15 DEBUG "metric": { 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG }, 12:18:15 DEBUG "instance": "10.142.0.51:9100" 12:18:15 DEBUG 12:18:15 DEBUG "value": [ 12:18:15 DEBUG }, 12:18:15 DEBUG [...] 12:18:15 DEBUG 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG "0.00043333333333445503" 12:18:15 DEBUG 12:18:15 DEBUG [...] 12:18:15 DEBUG ] 12:18:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:18:15 DEBUG 12:18:15 DEBUG } 12:18:15 DEBUG }, 12:18:15 DEBUG 12:18:15 DEBUG ] 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 12:18:15 DEBUG } 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:18:15 DEBUG } 12:18:15 DEBUG "0" 12:18:15 DEBUG }, 12:18:15 DEBUG ] 12:18:15 DEBUG "value": [ 12:18:15 DEBUG } 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG ] 12:18:15 DEBUG "0.029240066864752036" 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG { 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG "data": { 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG "result": [ 12:18:15 DEBUG { 12:18:15 DEBUG "metric": { 12:18:15 DEBUG "instance": "10.142.0.11:9100" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG [...] 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG "instance": "10.142.0.51:9100" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG "833284.6333333332" 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906695" 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG { 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG "data": { 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG "result": [ 12:18:15 DEBUG { 12:18:15 DEBUG "metric": { 12:18:15 DEBUG "instance": "10.142.0.11:9100" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG [...] 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG "instance": "10.142.0.51:9100" 12:18:15 DEBUG }, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG "19114.666666666664" 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906695" 12:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906695" 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG --- http response --- 12:18:15 INFO [http_cmd]: http status code OK 12:18:15 DEBUG --- status code --- 12:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:18:15 DEBUG --- http response --- 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:18:15 DEBUG { 12:18:15 DEBUG ----- output ----- 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG { 12:18:15 DEBUG "data": { 12:18:15 DEBUG "status": "success", 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG "data": { 12:18:15 DEBUG "result": [ 12:18:15 DEBUG "resultType": "vector", 12:18:15 DEBUG { 12:18:15 DEBUG "result": [ 12:18:15 DEBUG "metric": { 12:18:15 DEBUG { 12:18:15 DEBUG "pod": "ds-idrepo-0" 12:18:15 DEBUG "metric": { 12:18:15 DEBUG }, 12:18:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:18:15 DEBUG "value": [ 12:18:15 DEBUG }, 12:18:15 DEBUG 12:18:15 DEBUG "value": [ 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG [...] 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG [...] 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG 12:18:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:18:15 DEBUG 12:18:15 DEBUG }, 12:18:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:18:15 DEBUG "value": [ 12:18:15 DEBUG }, 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG "value": [ 12:18:15 DEBUG "758550528" 12:18:15 DEBUG 1735906695, 12:18:15 DEBUG ] 12:18:15 DEBUG "14989463552" 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 DEBUG ] 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:15 DEBUG } 12:18:20 INFO 12:18:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:18:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:18:20 INFO [loop_until]: OK (rc = 0) 12:18:20 DEBUG --- stdout --- 12:18:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 107m 0% 3856Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 82m 0% 19053Mi 66% 12:18:20 DEBUG --- stderr --- 12:18:20 DEBUG 12:18:24 INFO 12:18:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:18:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:18:24 INFO [loop_until]: OK (rc = 0) 12:18:24 DEBUG --- stdout --- 12:18:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 58m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:18:24 DEBUG --- stderr --- 12:18:24 DEBUG 12:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906753" 12:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906753" 12:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906753" 12:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906753" 12:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906753" 12:19:13 INFO [http_cmd]: http status code OK 12:19:13 DEBUG --- status code --- 12:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:13 DEBUG --- http response --- 12:19:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:13 INFO [http_cmd]: http status code OK 12:19:13 DEBUG --- status code --- 12:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:13 DEBUG --- http response --- 12:19:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:13 ERROR [http_cmd]: ERROR 12:19:13 ERROR --- status code --- 12:19:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:19:13 ERROR --- http response --- 12:19:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:19:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:19:13 INFO [http_cmd]: http status code OK 12:19:13 DEBUG --- status code --- 12:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:13 DEBUG --- http response --- 12:19:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:13 INFO [http_cmd]: http status code OK 12:19:13 DEBUG --- status code --- 12:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:13 DEBUG --- http response --- 12:19:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906754" 12:19:14 INFO [http_cmd]: http status code OK 12:19:14 DEBUG --- status code --- 12:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:14 DEBUG --- http response --- 12:19:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906754" 12:19:14 INFO [http_cmd]: http status code OK 12:19:14 DEBUG --- status code --- 12:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:14 DEBUG --- http response --- 12:19:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906754" 12:19:14 INFO [http_cmd]: http status code OK 12:19:14 DEBUG --- status code --- 12:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:14 DEBUG --- http response --- 12:19:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906754" 12:19:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906754" 12:19:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906754" 12:19:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906754" 12:19:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906754" 12:19:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906754" 12:19:14 INFO [http_cmd]: http status code OK 12:19:14 DEBUG --- status code --- 12:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:14 DEBUG --- http response --- 12:19:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:14 INFO [http_cmd]: http status code OK 12:19:14 DEBUG --- status code --- 12:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:14 DEBUG --- http response --- 12:19:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:14 INFO [http_cmd]: http status code OK 12:19:14 DEBUG --- status code --- 12:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:14 DEBUG --- http response --- 12:19:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:14 INFO [http_cmd]: http status code OK 12:19:14 INFO [http_cmd]: http status code OK 12:19:14 INFO [http_cmd]: http status code OK 12:19:14 DEBUG --- status code --- 12:19:14 DEBUG --- status code --- 12:19:14 DEBUG --- status code --- 12:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:14 DEBUG --- http response --- 12:19:14 DEBUG --- http response --- 12:19:14 DEBUG --- http response --- 12:19:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906755" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG --- http response --- 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG { 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG { 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG { 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG "data": { 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG "data": { 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "data": { 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "result": [ 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "result": [ 12:19:15 DEBUG { 12:19:15 DEBUG "result": [ 12:19:15 DEBUG { 12:19:15 DEBUG "metric": { 12:19:15 DEBUG { 12:19:15 DEBUG "metric": { 12:19:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:19:15 DEBUG "metric": { 12:19:15 DEBUG "instance": "10.142.0.11:9100" 12:19:15 DEBUG }, 12:19:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG [...] 12:19:15 DEBUG 12:19:15 DEBUG [...] 12:19:15 DEBUG 12:19:15 DEBUG [...] 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG "pod": "ds-cts-0" 12:19:15 DEBUG 12:19:15 DEBUG "instance": "10.142.0.51:9100" 12:19:15 DEBUG }, 12:19:15 DEBUG "pod": "ds-cts-0" 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG "405.0098879367173" 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG "1263037.2666666668" 12:19:15 DEBUG ] 12:19:15 DEBUG "0" 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906755" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG { 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG "data": { 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "result": [ 12:19:15 DEBUG { 12:19:15 DEBUG "metric": { 12:19:15 DEBUG "instance": "10.142.0.11:9100" 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG [...] 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG "instance": "10.142.0.51:9100" 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG "0.00033333333333397756" 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906755" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:19:15 DEBUG { 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG { 12:19:15 DEBUG "data": { 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "data": { 12:19:15 DEBUG "result": [ 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG { 12:19:15 DEBUG "result": [ 12:19:15 DEBUG "metric": { 12:19:15 DEBUG { 12:19:15 DEBUG "instance": "10.142.0.11:9100" 12:19:15 DEBUG "metric": { 12:19:15 DEBUG }, 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:19:15 DEBUG "value": [ 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG }, 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG "value": [ 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG [...] 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG [...] 12:19:15 DEBUG 12:19:15 DEBUG { 12:19:15 DEBUG 12:19:15 DEBUG "instance": "10.142.0.51:9100" 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG 12:19:15 DEBUG }, 12:19:15 DEBUG "data": { 12:19:15 DEBUG 12:19:15 DEBUG "value": [ 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG "result": [ 12:19:15 DEBUG }, 12:19:15 DEBUG "0.0004999999999995453" 12:19:15 DEBUG { 12:19:15 DEBUG "value": [ 12:19:15 DEBUG ] 12:19:15 DEBUG "metric": { 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG } 12:19:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:19:15 DEBUG "0.08785569257704198" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG ] 12:19:15 DEBUG }, 12:19:15 DEBUG ] 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG } 12:19:15 DEBUG "value": [ 12:19:15 DEBUG } 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG } 12:19:15 DEBUG 12:19:15 DEBUG ] 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG 12:19:15 DEBUG } 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:19:15 DEBUG 12:19:15 DEBUG } 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG [...] 12:19:15 DEBUG { 12:19:15 DEBUG 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG 12:19:15 DEBUG "data": { 12:19:15 DEBUG 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "pod": "ds-cts-0" 12:19:15 DEBUG "result": [ 12:19:15 DEBUG }, 12:19:15 DEBUG { 12:19:15 DEBUG "value": [ 12:19:15 DEBUG "metric": { 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG "instance": "10.142.0.11:9100" 12:19:15 DEBUG "0.00285649307844925" 12:19:15 DEBUG }, 12:19:15 DEBUG ] 12:19:15 DEBUG "value": [ 12:19:15 DEBUG } 12:19:15 DEBUG 12:19:15 DEBUG ] 12:19:15 DEBUG 12:19:15 DEBUG } 12:19:15 DEBUG 12:19:15 DEBUG } 12:19:15 DEBUG [...] 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG "instance": "10.142.0.51:9100" 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG "1246089.2666666664" 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906755" 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG { 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG "data": { 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "result": [ 12:19:15 DEBUG { 12:19:15 DEBUG "metric": { 12:19:15 DEBUG "instance": "10.142.0.11:9100" 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG [...] 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG "instance": "10.142.0.51:9100" 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG "31675.73333333333" 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906755" 12:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906755" 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 INFO [http_cmd]: http status code OK 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG --- status code --- 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG --- http response --- 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:19:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG ----- output ----- 12:19:15 DEBUG { 12:19:15 DEBUG { 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG "status": "success", 12:19:15 DEBUG "data": { 12:19:15 DEBUG "data": { 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "resultType": "vector", 12:19:15 DEBUG "result": [ 12:19:15 DEBUG "result": [ 12:19:15 DEBUG { 12:19:15 DEBUG { 12:19:15 DEBUG "metric": { 12:19:15 DEBUG "metric": { 12:19:15 DEBUG "pod": "ds-idrepo-0" 12:19:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:19:15 DEBUG }, 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG [...] 12:19:15 DEBUG [...] 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG 12:19:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:19:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:19:15 DEBUG }, 12:19:15 DEBUG }, 12:19:15 DEBUG "value": [ 12:19:15 DEBUG "value": [ 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG 1735906755, 12:19:15 DEBUG "758550528" 12:19:15 DEBUG "14988525568" 12:19:15 DEBUG ] 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 DEBUG ] 12:19:15 DEBUG ] 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:15 DEBUG } 12:19:20 INFO 12:19:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:19:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:19:20 INFO [loop_until]: OK (rc = 0) 12:19:20 DEBUG --- stdout --- 12:19:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 160m 1% 3856Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 125m 0% 19055Mi 67% 12:19:20 DEBUG --- stderr --- 12:19:20 DEBUG 12:19:24 INFO 12:19:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:19:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:19:24 INFO [loop_until]: OK (rc = 0) 12:19:24 DEBUG --- stdout --- 12:19:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 81m 13478Mi rcs-645c86fdd6-j4rsk 61m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:19:24 DEBUG --- stderr --- 12:19:24 DEBUG 12:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906813" 12:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906813" 12:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906813" 12:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906813" 12:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906813" 12:20:13 INFO [http_cmd]: http status code OK 12:20:13 ERROR [http_cmd]: ERROR 12:20:13 INFO [http_cmd]: http status code OK 12:20:13 INFO [http_cmd]: http status code OK 12:20:13 INFO [http_cmd]: http status code OK 12:20:13 DEBUG --- status code --- 12:20:13 ERROR --- status code --- 12:20:13 DEBUG --- status code --- 12:20:13 DEBUG --- status code --- 12:20:13 DEBUG --- status code --- 12:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:13 DEBUG --- http response --- 12:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:13 DEBUG --- http response --- 12:20:13 ERROR --- http response --- 12:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:13 DEBUG --- http response --- 12:20:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:20:13 DEBUG --- http response --- 12:20:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:20:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906814" 12:20:14 INFO [http_cmd]: http status code OK 12:20:14 DEBUG --- status code --- 12:20:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:14 DEBUG --- http response --- 12:20:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906814" 12:20:14 INFO [http_cmd]: http status code OK 12:20:14 DEBUG --- status code --- 12:20:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:14 DEBUG --- http response --- 12:20:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906814" 12:20:14 INFO [http_cmd]: http status code OK 12:20:14 DEBUG --- status code --- 12:20:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:14 DEBUG --- http response --- 12:20:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906814" 12:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906814" 12:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906814" 12:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906814" 12:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906814" 12:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906814" 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG --- status code --- 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG { 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG { 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG { 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG "data": { 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG "data": { 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG "data": { 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG "result": [ 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG "result": [ 12:20:15 DEBUG { 12:20:15 DEBUG "result": [ 12:20:15 DEBUG { 12:20:15 DEBUG "metric": { 12:20:15 DEBUG { 12:20:15 DEBUG "metric": { 12:20:15 DEBUG "pod": "ds-cts-0" 12:20:15 DEBUG "metric": { 12:20:15 DEBUG "instance": "10.142.0.11:9100" 12:20:15 DEBUG }, 12:20:15 DEBUG "pod": "ds-cts-0" 12:20:15 DEBUG }, 12:20:15 DEBUG "value": [ 12:20:15 DEBUG }, 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG [...] 12:20:15 DEBUG 12:20:15 DEBUG [...] 12:20:15 DEBUG 12:20:15 DEBUG [...] 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG "instance": "10.142.0.51:9100" 12:20:15 DEBUG }, 12:20:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:20:15 DEBUG 12:20:15 DEBUG "value": [ 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906815" 12:20:15 DEBUG }, 12:20:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG "value": [ 12:20:15 DEBUG }, 12:20:15 DEBUG "2201630.3" 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG "value": [ 12:20:15 DEBUG ] 12:20:15 DEBUG "0" 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG "0" 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG { 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG "data": { 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG "result": [ 12:20:15 DEBUG { 12:20:15 DEBUG "metric": { 12:20:15 DEBUG "instance": "10.142.0.11:9100" 12:20:15 DEBUG }, 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG [...] 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG "instance": "10.142.0.51:9100" 12:20:15 DEBUG }, 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG "0" 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG "data": { 12:20:15 DEBUG { 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG "result": [ 12:20:15 DEBUG "data": { 12:20:15 DEBUG { 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG "metric": { 12:20:15 DEBUG "result": [ 12:20:15 DEBUG "instance": "10.142.0.11:9100" 12:20:15 DEBUG { 12:20:15 DEBUG }, 12:20:15 DEBUG "metric": { 12:20:15 DEBUG "value": [ 12:20:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:20:15 DEBUG 12:20:15 DEBUG }, 12:20:15 DEBUG 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG [...] 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG [...] 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG "instance": "10.142.0.51:9100" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906815" 12:20:15 DEBUG 12:20:15 DEBUG }, 12:20:15 DEBUG 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG "value": [ 12:20:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG }, 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG "0.0005333333333320903" 12:20:15 DEBUG "value": [ 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG ] 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:20:15 DEBUG } 12:20:15 DEBUG "0.07965850384166229" 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG ] 12:20:15 DEBUG ] 12:20:15 DEBUG { 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 DEBUG "status": "success", 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG "data": { 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG } 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG } 12:20:15 DEBUG "result": [ 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG { 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:20:15 DEBUG "metric": { 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG "pod": "ds-cts-0" 12:20:15 DEBUG { 12:20:15 DEBUG }, 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG "value": [ 12:20:15 DEBUG "data": { 12:20:15 DEBUG 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG 12:20:15 DEBUG "result": [ 12:20:15 DEBUG 12:20:15 DEBUG { 12:20:15 DEBUG [...] 12:20:15 DEBUG "metric": { 12:20:15 DEBUG 12:20:15 DEBUG "instance": "10.142.0.11:9100" 12:20:15 DEBUG 12:20:15 DEBUG }, 12:20:15 DEBUG 12:20:15 DEBUG "value": [ 12:20:15 DEBUG "pod": "ds-idrepo-0" 12:20:15 DEBUG 12:20:15 DEBUG }, 12:20:15 DEBUG 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG [...] 12:20:15 DEBUG "0" 12:20:15 DEBUG 12:20:15 DEBUG ] 12:20:15 DEBUG 12:20:15 DEBUG } 12:20:15 DEBUG 12:20:15 DEBUG ] 12:20:15 DEBUG "instance": "10.142.0.51:9100" 12:20:15 DEBUG } 12:20:15 DEBUG }, 12:20:15 DEBUG } 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG "2184540.2666666666" 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG { 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG "data": { 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG "result": [ 12:20:15 DEBUG { 12:20:15 DEBUG "metric": { 12:20:15 DEBUG "instance": "10.142.0.11:9100" 12:20:15 DEBUG }, 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG [...] 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG "instance": "10.142.0.51:9100" 12:20:15 DEBUG }, 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG "20206.933333333334" 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906815" 12:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906815" 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG --- http response --- 12:20:15 INFO [http_cmd]: http status code OK 12:20:15 DEBUG --- status code --- 12:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:20:15 DEBUG --- http response --- 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:20:15 DEBUG { 12:20:15 DEBUG ----- output ----- 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG { 12:20:15 DEBUG "data": { 12:20:15 DEBUG "status": "success", 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG "data": { 12:20:15 DEBUG "result": [ 12:20:15 DEBUG "resultType": "vector", 12:20:15 DEBUG { 12:20:15 DEBUG "result": [ 12:20:15 DEBUG "metric": { 12:20:15 DEBUG { 12:20:15 DEBUG "pod": "ds-idrepo-0" 12:20:15 DEBUG "metric": { 12:20:15 DEBUG }, 12:20:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:20:15 DEBUG "value": [ 12:20:15 DEBUG }, 12:20:15 DEBUG 12:20:15 DEBUG "value": [ 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG [...] 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG [...] 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG 12:20:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:20:15 DEBUG 12:20:15 DEBUG }, 12:20:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:20:15 DEBUG "value": [ 12:20:15 DEBUG }, 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG "value": [ 12:20:15 DEBUG "758550528" 12:20:15 DEBUG 1735906815, 12:20:15 DEBUG ] 12:20:15 DEBUG "14991179776" 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 DEBUG ] 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:15 DEBUG } 12:20:20 INFO 12:20:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:20:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:20:20 INFO [loop_until]: OK (rc = 0) 12:20:20 DEBUG --- stdout --- 12:20:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 167m 1% 3861Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 724m 2% 19050Mi 66% 12:20:20 DEBUG --- stderr --- 12:20:20 DEBUG 12:20:24 INFO 12:20:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:20:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:20:24 INFO [loop_until]: OK (rc = 0) 12:20:24 DEBUG --- stdout --- 12:20:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 637m 13478Mi rcs-645c86fdd6-j4rsk 88m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:20:24 DEBUG --- stderr --- 12:20:24 DEBUG 12:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906873" 12:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906873" 12:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906873" 12:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906873" 12:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906873" 12:21:13 INFO [http_cmd]: http status code OK 12:21:13 DEBUG --- status code --- 12:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:13 DEBUG --- http response --- 12:21:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:13 INFO [http_cmd]: http status code OK 12:21:13 DEBUG --- status code --- 12:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:13 DEBUG --- http response --- 12:21:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:13 INFO [http_cmd]: http status code OK 12:21:13 DEBUG --- status code --- 12:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:13 DEBUG --- http response --- 12:21:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:13 ERROR [http_cmd]: ERROR 12:21:13 ERROR --- status code --- 12:21:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:21:13 ERROR --- http response --- 12:21:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:21:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:21:13 INFO [http_cmd]: http status code OK 12:21:13 DEBUG --- status code --- 12:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:13 DEBUG --- http response --- 12:21:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906874" 12:21:14 INFO [http_cmd]: http status code OK 12:21:14 DEBUG --- status code --- 12:21:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:14 DEBUG --- http response --- 12:21:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906874" 12:21:14 INFO [http_cmd]: http status code OK 12:21:14 DEBUG --- status code --- 12:21:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:14 DEBUG --- http response --- 12:21:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906874" 12:21:14 INFO [http_cmd]: http status code OK 12:21:14 DEBUG --- status code --- 12:21:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:14 DEBUG --- http response --- 12:21:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906875" 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 DEBUG --- status code --- 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- status code --- 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906875" 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG { 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG "data": { 12:21:15 DEBUG { 12:21:15 DEBUG { 12:21:15 DEBUG { 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG "result": [ 12:21:15 DEBUG "data": { 12:21:15 DEBUG "data": { 12:21:15 DEBUG "data": { 12:21:15 DEBUG { 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG "metric": { 12:21:15 DEBUG "result": [ 12:21:15 DEBUG "result": [ 12:21:15 DEBUG "result": [ 12:21:15 DEBUG "instance": "10.142.0.11:9100" 12:21:15 DEBUG { 12:21:15 DEBUG { 12:21:15 DEBUG { 12:21:15 DEBUG }, 12:21:15 DEBUG "metric": { 12:21:15 DEBUG "metric": { 12:21:15 DEBUG "metric": { 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "instance": "10.142.0.11:9100" 12:21:15 DEBUG "pod": "ds-cts-0" 12:21:15 DEBUG "pod": "ds-cts-0" 12:21:15 DEBUG 12:21:15 DEBUG }, 12:21:15 DEBUG }, 12:21:15 DEBUG }, 12:21:15 DEBUG 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "value": [ 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG [...] 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG [...] 12:21:15 DEBUG [...] 12:21:15 DEBUG [...] 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG "instance": "10.142.0.51:9100" 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG }, 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "instance": "10.142.0.51:9100" 12:21:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:21:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG }, 12:21:15 DEBUG }, 12:21:15 DEBUG }, 12:21:15 DEBUG "2399807.2333333325" 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "value": [ 12:21:15 DEBUG ] 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG } 12:21:15 DEBUG "0.00033333333333397756" 12:21:15 DEBUG "478.085789320105" 12:21:15 DEBUG "0" 12:21:15 DEBUG ] 12:21:15 DEBUG ] 12:21:15 DEBUG ] 12:21:15 DEBUG ] 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG ] 12:21:15 DEBUG } 12:21:15 DEBUG ] 12:21:15 DEBUG ] 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906875" 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- status code --- 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG ----- output ----- 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG { 12:21:15 DEBUG { 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG { 12:21:15 DEBUG "data": { 12:21:15 DEBUG "data": { 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG "data": { 12:21:15 DEBUG "result": [ 12:21:15 DEBUG "result": [ 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG { 12:21:15 DEBUG { 12:21:15 DEBUG { 12:21:15 DEBUG "result": [ 12:21:15 DEBUG "metric": { 12:21:15 DEBUG "metric": { 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG { 12:21:15 DEBUG "pod": "ds-cts-0" 12:21:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:21:15 DEBUG "data": { 12:21:15 DEBUG "metric": { 12:21:15 DEBUG }, 12:21:15 DEBUG }, 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG "instance": "10.142.0.11:9100" 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "result": [ 12:21:15 DEBUG }, 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG { 12:21:15 DEBUG "value": [ 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG "metric": { 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG "instance": "10.142.0.11:9100" 12:21:15 DEBUG 12:21:15 DEBUG [...] 12:21:15 DEBUG [...] 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG }, 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG "value": [ 12:21:15 DEBUG [...] 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:21:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG }, 12:21:15 DEBUG }, 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG [...] 12:21:15 DEBUG "instance": "10.142.0.51:9100" 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "value": [ 12:21:15 DEBUG { 12:21:15 DEBUG 12:21:15 DEBUG }, 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "0.09718911584475375" 12:21:15 DEBUG "0.666223790057146" 12:21:15 DEBUG "data": { 12:21:15 DEBUG 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG ] 12:21:15 DEBUG ] 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG "instance": "10.142.0.51:9100" 12:21:15 DEBUG "0.00036666666666557526" 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG "result": [ 12:21:15 DEBUG }, 12:21:15 DEBUG ] 12:21:15 DEBUG ] 12:21:15 DEBUG ] 12:21:15 DEBUG { 12:21:15 DEBUG "value": [ 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG "metric": { 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG ] 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 DEBUG "instance": "10.142.0.11:9100" 12:21:15 DEBUG "2382842.0666666673" 12:21:15 DEBUG } 12:21:15 DEBUG }, 12:21:15 DEBUG ] 12:21:15 DEBUG } 12:21:15 DEBUG "value": [ 12:21:15 DEBUG } 12:21:15 DEBUG 12:21:15 DEBUG ] 12:21:15 DEBUG 12:21:15 DEBUG } 12:21:15 DEBUG 12:21:15 DEBUG } 12:21:15 DEBUG [...] 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG 12:21:15 DEBUG "instance": "10.142.0.51:9100" 12:21:15 DEBUG }, 12:21:15 DEBUG "value": [ 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG "18022.399999999998" 12:21:15 DEBUG ] 12:21:15 DEBUG } 12:21:15 DEBUG ] 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906875" 12:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906875" 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG { 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG "data": { 12:21:15 INFO [http_cmd]: http status code OK 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG --- status code --- 12:21:15 DEBUG "result": [ 12:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:21:15 DEBUG { 12:21:15 DEBUG --- http response --- 12:21:15 DEBUG "metric": { 12:21:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:21:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:21:15 DEBUG ----- output ----- 12:21:15 DEBUG }, 12:21:15 DEBUG { 12:21:15 DEBUG "value": [ 12:21:15 DEBUG "status": "success", 12:21:15 DEBUG 12:21:15 DEBUG "data": { 12:21:15 DEBUG 12:21:15 DEBUG "resultType": "vector", 12:21:15 DEBUG 12:21:15 DEBUG "result": [ 12:21:15 DEBUG [...] 12:21:15 DEBUG { 12:21:15 DEBUG 12:21:15 DEBUG "metric": { 12:21:15 DEBUG 12:21:15 DEBUG "pod": "ds-idrepo-0" 12:21:15 DEBUG 12:21:15 DEBUG }, 12:21:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:21:15 DEBUG "value": [ 12:21:15 DEBUG }, 12:21:15 DEBUG 12:21:15 DEBUG "value": [ 12:21:15 DEBUG 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG 12:21:15 DEBUG "14988226560" 12:21:15 DEBUG [...] 12:21:15 DEBUG ] 12:21:15 DEBUG 12:21:15 DEBUG } 12:21:15 DEBUG 12:21:15 DEBUG ] 12:21:15 DEBUG 12:21:15 DEBUG } 12:21:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:21:15 DEBUG } 12:21:15 DEBUG }, 12:21:15 DEBUG "value": [ 12:21:15 DEBUG 1735906875, 12:21:15 DEBUG "758562816" 12:21:15 DEBUG ] 12:21:15 DEBUG } 12:21:15 DEBUG ] 12:21:15 DEBUG } 12:21:15 DEBUG } 12:21:20 INFO 12:21:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:21:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:21:20 INFO [loop_until]: OK (rc = 0) 12:21:20 DEBUG --- stdout --- 12:21:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 216m 1% 3854Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 183m 0% 19053Mi 66% 12:21:20 DEBUG --- stderr --- 12:21:20 DEBUG 12:21:24 INFO 12:21:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:21:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:21:24 INFO [loop_until]: OK (rc = 0) 12:21:24 DEBUG --- stdout --- 12:21:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 174m 13479Mi rcs-645c86fdd6-j4rsk 205m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:21:24 DEBUG --- stderr --- 12:21:24 DEBUG 12:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906933" 12:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906933" 12:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906933" 12:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906933" 12:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906933" 12:22:13 INFO [http_cmd]: http status code OK 12:22:13 DEBUG --- status code --- 12:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:13 DEBUG --- http response --- 12:22:13 ERROR [http_cmd]: ERROR 12:22:13 ERROR --- status code --- 12:22:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:22:13 ERROR --- http response --- 12:22:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:13 INFO [http_cmd]: http status code OK 12:22:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:22:13 DEBUG --- status code --- 12:22:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:13 DEBUG --- http response --- 12:22:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:13 INFO [http_cmd]: http status code OK 12:22:13 DEBUG --- status code --- 12:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:13 INFO [http_cmd]: http status code OK 12:22:13 DEBUG --- http response --- 12:22:13 DEBUG --- status code --- 12:22:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:13 DEBUG --- http response --- 12:22:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906934" 12:22:14 INFO [http_cmd]: http status code OK 12:22:14 DEBUG --- status code --- 12:22:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:14 DEBUG --- http response --- 12:22:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906934" 12:22:14 INFO [http_cmd]: http status code OK 12:22:14 DEBUG --- status code --- 12:22:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:14 DEBUG --- http response --- 12:22:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906934" 12:22:14 INFO [http_cmd]: http status code OK 12:22:14 DEBUG --- status code --- 12:22:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:14 DEBUG --- http response --- 12:22:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG --- http response --- 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG --- http response --- 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG { 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG { 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:22:15 DEBUG "data": { 12:22:15 DEBUG { 12:22:15 DEBUG "data": { 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG { 12:22:15 DEBUG "result": [ 12:22:15 DEBUG "data": { 12:22:15 DEBUG "result": [ 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG { 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG { 12:22:15 DEBUG "data": { 12:22:15 DEBUG "metric": { 12:22:15 DEBUG "result": [ 12:22:15 DEBUG "metric": { 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG "pod": "ds-cts-0" 12:22:15 DEBUG { 12:22:15 DEBUG "instance": "10.142.0.11:9100" 12:22:15 DEBUG "result": [ 12:22:15 DEBUG }, 12:22:15 DEBUG "metric": { 12:22:15 DEBUG }, 12:22:15 DEBUG { 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "instance": "10.142.0.11:9100" 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "metric": { 12:22:15 DEBUG 12:22:15 DEBUG }, 12:22:15 DEBUG 12:22:15 DEBUG "pod": "ds-cts-0" 12:22:15 DEBUG 12:22:15 DEBUG "value": [ 12:22:15 DEBUG 12:22:15 DEBUG }, 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG "value": [ 12:22:15 DEBUG [...] 12:22:15 DEBUG 12:22:15 DEBUG [...] 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG [...] 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG [...] 12:22:15 DEBUG "pod": "ds-idrepo-0" 12:22:15 DEBUG 12:22:15 DEBUG "instance": "10.142.0.51:9100" 12:22:15 DEBUG 12:22:15 DEBUG }, 12:22:15 DEBUG 12:22:15 DEBUG }, 12:22:15 DEBUG 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "instance": "10.142.0.51:9100" 12:22:15 DEBUG "value": [ 12:22:15 DEBUG 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG }, 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG "pod": "ds-idrepo-0" 12:22:15 DEBUG "0" 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "1606789.6000000008" 12:22:15 DEBUG }, 12:22:15 DEBUG ] 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG ] 12:22:15 DEBUG "value": [ 12:22:15 DEBUG } 12:22:15 DEBUG "0.000666666666666534" 12:22:15 DEBUG } 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG ] 12:22:15 DEBUG ] 12:22:15 DEBUG ] 12:22:15 DEBUG "747.347442590942" 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG ] 12:22:15 DEBUG } 12:22:15 DEBUG ] 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG ] 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906935" 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG { 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:22:15 DEBUG "data": { 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG { 12:22:15 DEBUG "result": [ 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG { 12:22:15 DEBUG "data": { 12:22:15 DEBUG "metric": { 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:22:15 DEBUG "result": [ 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG }, 12:22:15 DEBUG { 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "metric": { 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG 12:22:15 DEBUG "pod": "ds-cts-0" 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG 12:22:15 DEBUG }, 12:22:15 DEBUG 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:22:15 DEBUG "value": [ 12:22:15 DEBUG [...] 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG { 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG "data": { 12:22:15 DEBUG [...] 12:22:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG 12:22:15 DEBUG }, 12:22:15 DEBUG "result": [ 12:22:15 DEBUG 12:22:15 DEBUG "value": [ 12:22:15 DEBUG { 12:22:15 DEBUG 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG "metric": { 12:22:15 DEBUG "pod": "ds-idrepo-0" 12:22:15 DEBUG "0.13267566867770025" 12:22:15 DEBUG "instance": "10.142.0.11:9100" 12:22:15 DEBUG }, 12:22:15 DEBUG "value": [ 12:22:15 DEBUG }, 12:22:15 DEBUG ] 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG "value": [ 12:22:15 DEBUG } 12:22:15 DEBUG "0.0033102028398440788" 12:22:15 DEBUG 12:22:15 DEBUG ] 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG ] 12:22:15 DEBUG 12:22:15 DEBUG } 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG } 12:22:15 DEBUG 12:22:15 DEBUG } 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG ] 12:22:15 DEBUG [...] 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG } 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG } 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:22:15 DEBUG 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG "instance": "10.142.0.51:9100" 12:22:15 DEBUG { 12:22:15 DEBUG { 12:22:15 DEBUG }, 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "data": { 12:22:15 DEBUG "data": { 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG "0.0012999999999976808" 12:22:15 DEBUG "result": [ 12:22:15 DEBUG "result": [ 12:22:15 DEBUG ] 12:22:15 DEBUG { 12:22:15 DEBUG { 12:22:15 DEBUG } 12:22:15 DEBUG "metric": { 12:22:15 DEBUG "metric": { 12:22:15 DEBUG ] 12:22:15 DEBUG "instance": "10.142.0.11:9100" 12:22:15 DEBUG "instance": "10.142.0.11:9100" 12:22:15 DEBUG } 12:22:15 DEBUG }, 12:22:15 DEBUG }, 12:22:15 DEBUG } 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "value": [ 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG [...] 12:22:15 DEBUG [...] 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG "instance": "10.142.0.51:9100" 12:22:15 DEBUG "instance": "10.142.0.51:9100" 12:22:15 DEBUG }, 12:22:15 DEBUG }, 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "value": [ 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG "14062.933333333332" 12:22:15 DEBUG "1592361.966666666" 12:22:15 DEBUG ] 12:22:15 DEBUG ] 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG ] 12:22:15 DEBUG ] 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906935" 12:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906935" 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 INFO [http_cmd]: http status code OK 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG --- status code --- 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG --- http response --- 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:22:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG ----- output ----- 12:22:15 DEBUG { 12:22:15 DEBUG { 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG "status": "success", 12:22:15 DEBUG "data": { 12:22:15 DEBUG "data": { 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG "resultType": "vector", 12:22:15 DEBUG "result": [ 12:22:15 DEBUG "result": [ 12:22:15 DEBUG { 12:22:15 DEBUG { 12:22:15 DEBUG "metric": { 12:22:15 DEBUG "metric": { 12:22:15 DEBUG "pod": "ds-idrepo-0" 12:22:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:22:15 DEBUG }, 12:22:15 DEBUG }, 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "value": [ 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG [...] 12:22:15 DEBUG [...] 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG 12:22:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:22:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:22:15 DEBUG }, 12:22:15 DEBUG }, 12:22:15 DEBUG "value": [ 12:22:15 DEBUG "value": [ 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG 1735906935, 12:22:15 DEBUG "14987165696" 12:22:15 DEBUG "758558720" 12:22:15 DEBUG ] 12:22:15 DEBUG ] 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG ] 12:22:15 DEBUG ] 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:15 DEBUG } 12:22:20 INFO 12:22:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:22:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:22:20 INFO [loop_until]: OK (rc = 0) 12:22:20 DEBUG --- stdout --- 12:22:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 216m 1% 3856Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 190m 0% 19055Mi 67% 12:22:20 DEBUG --- stderr --- 12:22:20 DEBUG 12:22:24 INFO 12:22:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:22:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:22:24 INFO [loop_until]: OK (rc = 0) 12:22:24 DEBUG --- stdout --- 12:22:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 12m 13479Mi rcs-645c86fdd6-j4rsk 66m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:22:24 DEBUG --- stderr --- 12:22:24 DEBUG 12:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906993" 12:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735906993" 12:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906993" 12:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906993" 12:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906993" 12:23:13 INFO [http_cmd]: http status code OK 12:23:13 DEBUG --- status code --- 12:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:13 DEBUG --- http response --- 12:23:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:13 INFO [http_cmd]: http status code OK 12:23:13 INFO [http_cmd]: http status code OK 12:23:13 INFO [http_cmd]: http status code OK 12:23:13 DEBUG --- status code --- 12:23:13 ERROR [http_cmd]: ERROR 12:23:13 DEBUG --- status code --- 12:23:13 DEBUG --- status code --- 12:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:13 ERROR --- status code --- 12:23:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:13 DEBUG --- http response --- 12:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:13 ERROR --- http response --- 12:23:13 DEBUG --- http response --- 12:23:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:13 DEBUG --- http response --- 12:23:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:23:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:23:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906994" 12:23:14 INFO [http_cmd]: http status code OK 12:23:14 DEBUG --- status code --- 12:23:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:14 DEBUG --- http response --- 12:23:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906994" 12:23:14 INFO [http_cmd]: http status code OK 12:23:14 DEBUG --- status code --- 12:23:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:14 DEBUG --- http response --- 12:23:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906994" 12:23:14 INFO [http_cmd]: http status code OK 12:23:14 DEBUG --- status code --- 12:23:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:14 DEBUG --- http response --- 12:23:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- http response --- 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906995" 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG { 12:23:15 DEBUG "status": "success", 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG "data": { 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG "result": [ 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG { 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:23:15 DEBUG "metric": { 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG { 12:23:15 DEBUG }, 12:23:15 DEBUG { 12:23:15 DEBUG { 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG "value": [ 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG "data": { 12:23:15 DEBUG 12:23:15 DEBUG "data": { 12:23:15 DEBUG "data": { 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG "result": [ 12:23:15 DEBUG 12:23:15 DEBUG "result": [ 12:23:15 DEBUG "result": [ 12:23:15 DEBUG { 12:23:15 DEBUG [...] 12:23:15 DEBUG { 12:23:15 DEBUG { 12:23:15 DEBUG "metric": { 12:23:15 DEBUG 12:23:15 DEBUG "metric": { 12:23:15 DEBUG "metric": { 12:23:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:23:15 DEBUG 12:23:15 DEBUG "instance": "10.142.0.11:9100" 12:23:15 DEBUG "instance": "10.142.0.11:9100" 12:23:15 DEBUG }, 12:23:15 DEBUG 12:23:15 DEBUG }, 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG "pod": "ds-cts-0" 12:23:15 DEBUG "value": [ 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 12:23:15 DEBUG }, 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG [...] 12:23:15 DEBUG "0" 12:23:15 DEBUG [...] 12:23:15 DEBUG [...] 12:23:15 DEBUG 12:23:15 DEBUG ] 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG } 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG ] 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG "pod": "ds-cts-0" 12:23:15 DEBUG } 12:23:15 DEBUG "instance": "10.142.0.51:9100" 12:23:15 DEBUG "instance": "10.142.0.51:9100" 12:23:15 DEBUG }, 12:23:15 DEBUG } 12:23:15 DEBUG }, 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG "value": [ 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG "565.4724925795541" 12:23:15 DEBUG "0.0003333333333330302" 12:23:15 DEBUG "805258.8666666667" 12:23:15 DEBUG ] 12:23:15 DEBUG ] 12:23:15 DEBUG ] 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG ] 12:23:15 DEBUG ] 12:23:15 DEBUG ] 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735906995" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG { 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG "data": { 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG "result": [ 12:23:15 DEBUG { 12:23:15 DEBUG "metric": { 12:23:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG [...] 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG "pod": "ds-cts-0" 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG "0.002753779250362661" 12:23:15 DEBUG ] 12:23:15 DEBUG } 12:23:15 DEBUG ] 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG { 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG "data": { 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG "result": [ 12:23:15 DEBUG { 12:23:15 DEBUG "metric": { 12:23:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG [...] 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:23:15 DEBUG }, 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG "value": [ 12:23:15 DEBUG { 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG "0.14209327774268526" 12:23:15 DEBUG "data": { 12:23:15 DEBUG ] 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG } 12:23:15 DEBUG "result": [ 12:23:15 DEBUG ] 12:23:15 DEBUG { 12:23:15 DEBUG } 12:23:15 DEBUG "metric": { 12:23:15 DEBUG } 12:23:15 DEBUG "instance": "10.142.0.11:9100" 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG [...] 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG "instance": "10.142.0.51:9100" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG }, 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG "value": [ 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG "0.00046666666666700016" 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:23:15 DEBUG ] 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG } 12:23:15 DEBUG { 12:23:15 DEBUG { 12:23:15 DEBUG ] 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG "data": { 12:23:15 DEBUG "data": { 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG "result": [ 12:23:15 DEBUG "result": [ 12:23:15 DEBUG { 12:23:15 DEBUG { 12:23:15 DEBUG "metric": { 12:23:15 DEBUG "metric": { 12:23:15 DEBUG "instance": "10.142.0.11:9100" 12:23:15 DEBUG "instance": "10.142.0.11:9100" 12:23:15 DEBUG }, 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG [...] 12:23:15 DEBUG [...] 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG 12:23:15 DEBUG "instance": "10.142.0.51:9100" 12:23:15 DEBUG "instance": "10.142.0.51:9100" 12:23:15 DEBUG }, 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG "22937.6" 12:23:15 DEBUG "788643.1" 12:23:15 DEBUG ] 12:23:15 DEBUG ] 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG ] 12:23:15 DEBUG ] 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735906995" 12:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735906995" 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG { 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG "data": { 12:23:15 DEBUG "resultType": "vector", 12:23:15 INFO [http_cmd]: http status code OK 12:23:15 DEBUG "result": [ 12:23:15 DEBUG --- status code --- 12:23:15 DEBUG { 12:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:23:15 DEBUG "metric": { 12:23:15 DEBUG --- http response --- 12:23:15 DEBUG "pod": "ds-idrepo-0" 12:23:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:23:15 DEBUG }, 12:23:15 DEBUG ----- output ----- 12:23:15 DEBUG "value": [ 12:23:15 DEBUG { 12:23:15 DEBUG 12:23:15 DEBUG "status": "success", 12:23:15 DEBUG 12:23:15 DEBUG "data": { 12:23:15 DEBUG 12:23:15 DEBUG "resultType": "vector", 12:23:15 DEBUG [...] 12:23:15 DEBUG "result": [ 12:23:15 DEBUG 12:23:15 DEBUG { 12:23:15 DEBUG 12:23:15 DEBUG "metric": { 12:23:15 DEBUG 12:23:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:23:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:23:15 DEBUG }, 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG 12:23:15 DEBUG "758566912" 12:23:15 DEBUG 12:23:15 DEBUG ] 12:23:15 DEBUG [...] 12:23:15 DEBUG } 12:23:15 DEBUG 12:23:15 DEBUG ] 12:23:15 DEBUG 12:23:15 DEBUG } 12:23:15 DEBUG 12:23:15 DEBUG } 12:23:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:23:15 DEBUG }, 12:23:15 DEBUG "value": [ 12:23:15 DEBUG 1735906995, 12:23:15 DEBUG "14990098432" 12:23:15 DEBUG ] 12:23:15 DEBUG } 12:23:15 DEBUG ] 12:23:15 DEBUG } 12:23:15 DEBUG } 12:23:20 INFO 12:23:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:23:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:23:20 INFO [loop_until]: OK (rc = 0) 12:23:20 DEBUG --- stdout --- 12:23:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 207m 1% 3864Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 170m 0% 19056Mi 67% 12:23:20 DEBUG --- stderr --- 12:23:20 DEBUG 12:23:24 INFO 12:23:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:23:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:23:24 INFO [loop_until]: OK (rc = 0) 12:23:24 DEBUG --- stdout --- 12:23:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 49m 13478Mi rcs-645c86fdd6-j4rsk 14m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:23:24 DEBUG --- stderr --- 12:23:24 DEBUG 12:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907053" 12:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907053" 12:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907053" 12:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907053" 12:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907053" 12:24:14 INFO [http_cmd]: http status code OK 12:24:14 DEBUG --- status code --- 12:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:14 DEBUG --- http response --- 12:24:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:14 ERROR [http_cmd]: ERROR 12:24:14 ERROR --- status code --- 12:24:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:24:14 ERROR --- http response --- 12:24:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:24:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:24:14 INFO [http_cmd]: http status code OK 12:24:14 DEBUG --- status code --- 12:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:14 DEBUG --- http response --- 12:24:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:14 INFO [http_cmd]: http status code OK 12:24:14 DEBUG --- status code --- 12:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:14 DEBUG --- http response --- 12:24:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:14 INFO [http_cmd]: http status code OK 12:24:14 DEBUG --- status code --- 12:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:14 DEBUG --- http response --- 12:24:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907054" 12:24:14 INFO [http_cmd]: http status code OK 12:24:14 DEBUG --- status code --- 12:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:14 DEBUG --- http response --- 12:24:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907054" 12:24:14 INFO [http_cmd]: http status code OK 12:24:14 DEBUG --- status code --- 12:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:14 DEBUG --- http response --- 12:24:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907054" 12:24:14 INFO [http_cmd]: http status code OK 12:24:14 DEBUG --- status code --- 12:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:14 DEBUG --- http response --- 12:24:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907055" 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG { 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG "data": { 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG "result": [ 12:24:15 DEBUG { 12:24:15 DEBUG "metric": { 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG "pod": "ds-cts-0" 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG }, 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG "value": [ 12:24:15 DEBUG { 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:24:15 DEBUG 12:24:15 DEBUG "data": { 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG { 12:24:15 DEBUG [...] 12:24:15 DEBUG 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG 12:24:15 DEBUG "result": [ 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG "data": { 12:24:15 DEBUG 12:24:15 DEBUG { 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:24:15 DEBUG "metric": { 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG "result": [ 12:24:15 DEBUG }, 12:24:15 DEBUG "pod": "ds-cts-0" 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:24:15 DEBUG { 12:24:15 DEBUG "value": [ 12:24:15 DEBUG }, 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG "metric": { 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG { 12:24:15 DEBUG "instance": "10.142.0.11:9100" 12:24:15 DEBUG "0" 12:24:15 DEBUG 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG }, 12:24:15 DEBUG ] 12:24:15 DEBUG 12:24:15 DEBUG "data": { 12:24:15 DEBUG "value": [ 12:24:15 DEBUG } 12:24:15 DEBUG 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG [...] 12:24:15 DEBUG ] 12:24:15 DEBUG [...] 12:24:15 DEBUG "result": [ 12:24:15 DEBUG 12:24:15 DEBUG } 12:24:15 DEBUG 12:24:15 DEBUG { 12:24:15 DEBUG 12:24:15 DEBUG } 12:24:15 DEBUG 12:24:15 DEBUG "metric": { 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG "instance": "10.142.0.11:9100" 12:24:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:24:15 DEBUG "instance": "10.142.0.51:9100" 12:24:15 DEBUG }, 12:24:15 DEBUG }, 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG "value": [ 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG 12:24:15 DEBUG "0" 12:24:15 DEBUG "0.00033333333333397756" 12:24:15 DEBUG 12:24:15 DEBUG ] 12:24:15 DEBUG ] 12:24:15 DEBUG [...] 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 DEBUG 12:24:15 DEBUG ] 12:24:15 DEBUG ] 12:24:15 DEBUG 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 DEBUG 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 DEBUG "instance": "10.142.0.51:9100" 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG "95034.43333333333" 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907055" 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG { 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG "data": { 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG "result": [ 12:24:15 DEBUG { 12:24:15 DEBUG "metric": { 12:24:15 DEBUG "pod": "ds-cts-0" 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG [...] 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG "0.0003478543926535302" 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG { 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG "data": { 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG "result": [ 12:24:15 DEBUG { 12:24:15 DEBUG "metric": { 12:24:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG [...] 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG "0.11360075817216794" 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG { 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG "data": { 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG "result": [ 12:24:15 DEBUG { 12:24:15 DEBUG "metric": { 12:24:15 DEBUG "instance": "10.142.0.11:9100" 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG [...] 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG "instance": "10.142.0.51:9100" 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG "0.0007333333333330454" 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG { 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG "data": { 12:24:15 DEBUG { 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG "result": [ 12:24:15 DEBUG "data": { 12:24:15 DEBUG { 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG "metric": { 12:24:15 DEBUG "result": [ 12:24:15 DEBUG "instance": "10.142.0.11:9100" 12:24:15 DEBUG { 12:24:15 DEBUG }, 12:24:15 DEBUG "metric": { 12:24:15 DEBUG "value": [ 12:24:15 DEBUG "instance": "10.142.0.11:9100" 12:24:15 DEBUG 12:24:15 DEBUG }, 12:24:15 DEBUG 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG [...] 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG [...] 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG "instance": "10.142.0.51:9100" 12:24:15 DEBUG 12:24:15 DEBUG }, 12:24:15 DEBUG 12:24:15 DEBUG "value": [ 12:24:15 DEBUG "instance": "10.142.0.51:9100" 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG }, 12:24:15 DEBUG "78447.23333333334" 12:24:15 DEBUG "value": [ 12:24:15 DEBUG ] 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG } 12:24:15 DEBUG "28125.866666666665" 12:24:15 DEBUG ] 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907055" 12:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907055" 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 INFO [http_cmd]: http status code OK 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG --- status code --- 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG --- http response --- 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:24:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG ----- output ----- 12:24:15 DEBUG { 12:24:15 DEBUG { 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG "status": "success", 12:24:15 DEBUG "data": { 12:24:15 DEBUG "data": { 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG "resultType": "vector", 12:24:15 DEBUG "result": [ 12:24:15 DEBUG "result": [ 12:24:15 DEBUG { 12:24:15 DEBUG { 12:24:15 DEBUG "metric": { 12:24:15 DEBUG "metric": { 12:24:15 DEBUG "pod": "ds-idrepo-0" 12:24:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:24:15 DEBUG }, 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG [...] 12:24:15 DEBUG [...] 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG 12:24:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:24:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:24:15 DEBUG }, 12:24:15 DEBUG }, 12:24:15 DEBUG "value": [ 12:24:15 DEBUG "value": [ 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG 1735907055, 12:24:15 DEBUG "758558720" 12:24:15 DEBUG "14990155776" 12:24:15 DEBUG ] 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 DEBUG ] 12:24:15 DEBUG ] 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:15 DEBUG } 12:24:20 INFO 12:24:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:24:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:24:20 INFO [loop_until]: OK (rc = 0) 12:24:20 DEBUG --- stdout --- 12:24:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 156m 0% 3865Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 160m 0% 19056Mi 67% 12:24:20 DEBUG --- stderr --- 12:24:20 DEBUG 12:24:24 INFO 12:24:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:24:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:24:24 INFO [loop_until]: OK (rc = 0) 12:24:24 DEBUG --- stdout --- 12:24:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 44m 13478Mi rcs-645c86fdd6-j4rsk 40m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:24:24 DEBUG --- stderr --- 12:24:24 DEBUG 12:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907114" 12:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907114" 12:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907114" 12:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907114" 12:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907114" 12:25:14 ERROR [http_cmd]: ERROR 12:25:14 ERROR --- status code --- 12:25:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:25:14 ERROR --- http response --- 12:25:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:25:14 INFO [http_cmd]: http status code OK 12:25:14 INFO [http_cmd]: http status code OK 12:25:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:25:14 INFO [http_cmd]: http status code OK 12:25:14 DEBUG --- status code --- 12:25:14 INFO [http_cmd]: http status code OK 12:25:14 DEBUG --- status code --- 12:25:14 DEBUG --- status code --- 12:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:14 DEBUG --- status code --- 12:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:14 DEBUG --- http response --- 12:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:14 DEBUG --- http response --- 12:25:14 DEBUG --- http response --- 12:25:14 DEBUG --- http response --- 12:25:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907114" 12:25:14 INFO [http_cmd]: http status code OK 12:25:14 DEBUG --- status code --- 12:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:14 DEBUG --- http response --- 12:25:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907114" 12:25:14 INFO [http_cmd]: http status code OK 12:25:14 DEBUG --- status code --- 12:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:14 DEBUG --- http response --- 12:25:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907114" 12:25:14 INFO [http_cmd]: http status code OK 12:25:14 DEBUG --- status code --- 12:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:14 DEBUG --- http response --- 12:25:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- http response --- 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG { 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG "data": { 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG "result": [ 12:25:15 DEBUG { 12:25:15 DEBUG "metric": { 12:25:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:25:15 DEBUG }, 12:25:15 DEBUG "value": [ 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG [...] 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG "pod": "ds-idrepo-0" 12:25:15 DEBUG }, 12:25:15 DEBUG "value": [ 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG "722.0832847948801" 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG "data": { 12:25:15 DEBUG { 12:25:15 DEBUG { 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG "result": [ 12:25:15 DEBUG "data": { 12:25:15 DEBUG "data": { 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG { 12:25:15 DEBUG "result": [ 12:25:15 DEBUG "result": [ 12:25:15 DEBUG "metric": { 12:25:15 DEBUG { 12:25:15 DEBUG { 12:25:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:25:15 DEBUG "metric": { 12:25:15 DEBUG "metric": { 12:25:15 DEBUG }, 12:25:15 DEBUG "instance": "10.142.0.11:9100" 12:25:15 DEBUG "instance": "10.142.0.11:9100" 12:25:15 DEBUG "value": [ 12:25:15 DEBUG }, 12:25:15 DEBUG }, 12:25:15 DEBUG 12:25:15 DEBUG "value": [ 12:25:15 DEBUG "value": [ 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG [...] 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG [...] 12:25:15 DEBUG [...] 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG "pod": "ds-idrepo-0" 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG }, 12:25:15 DEBUG "instance": "10.142.0.51:9100" 12:25:15 DEBUG "instance": "10.142.0.51:9100" 12:25:15 DEBUG "value": [ 12:25:15 DEBUG }, 12:25:15 DEBUG }, 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG "value": [ 12:25:15 DEBUG "value": [ 12:25:15 DEBUG "323596.46634307696" 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG ] 12:25:15 DEBUG "0.0003333333333330302" 12:25:15 DEBUG "93149.4" 12:25:15 DEBUG } 12:25:15 DEBUG ] 12:25:15 DEBUG ] 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 DEBUG ] 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG { 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG "data": { 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG "result": [ 12:25:15 DEBUG { 12:25:15 DEBUG "metric": { 12:25:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:25:15 DEBUG }, 12:25:15 DEBUG "value": [ 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG [...] 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG "pod": "ds-cts-0" 12:25:15 DEBUG }, 12:25:15 DEBUG "value": [ 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG "0" 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG { 12:25:15 DEBUG "status": "success", 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907115" 12:25:15 DEBUG "data": { 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG "result": [ 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG { 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG "metric": { 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:25:15 DEBUG }, 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG "value": [ 12:25:15 DEBUG { 12:25:15 DEBUG 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG 12:25:15 DEBUG "data": { 12:25:15 DEBUG 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG [...] 12:25:15 DEBUG "result": [ 12:25:15 DEBUG 12:25:15 DEBUG { 12:25:15 DEBUG 12:25:15 DEBUG "metric": { 12:25:15 DEBUG 12:25:15 DEBUG "instance": "10.142.0.11:9100" 12:25:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:25:15 DEBUG }, 12:25:15 DEBUG }, 12:25:15 DEBUG "value": [ 12:25:15 DEBUG "value": [ 12:25:15 DEBUG 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG 12:25:15 DEBUG "0.10417121444913933" 12:25:15 DEBUG 12:25:15 DEBUG ] 12:25:15 DEBUG [...] 12:25:15 DEBUG } 12:25:15 DEBUG 12:25:15 DEBUG ] 12:25:15 DEBUG 12:25:15 DEBUG } 12:25:15 DEBUG 12:25:15 DEBUG } 12:25:15 DEBUG "instance": "10.142.0.51:9100" 12:25:15 DEBUG }, 12:25:15 DEBUG "value": [ 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG "0.0005666666666636881" 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG { 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG "data": { 12:25:15 DEBUG { 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG "result": [ 12:25:15 DEBUG "data": { 12:25:15 DEBUG { 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG "metric": { 12:25:15 DEBUG "result": [ 12:25:15 DEBUG "instance": "10.142.0.11:9100" 12:25:15 DEBUG { 12:25:15 DEBUG }, 12:25:15 DEBUG "metric": { 12:25:15 DEBUG "value": [ 12:25:15 DEBUG "instance": "10.142.0.11:9100" 12:25:15 DEBUG 12:25:15 DEBUG }, 12:25:15 DEBUG 12:25:15 DEBUG "value": [ 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG [...] 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG [...] 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG "instance": "10.142.0.51:9100" 12:25:15 DEBUG 12:25:15 DEBUG }, 12:25:15 DEBUG 12:25:15 DEBUG "value": [ 12:25:15 DEBUG "instance": "10.142.0.51:9100" 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG }, 12:25:15 DEBUG "76110" 12:25:15 DEBUG "value": [ 12:25:15 DEBUG ] 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG } 12:25:15 DEBUG "20206.93333333333" 12:25:15 DEBUG ] 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907115" 12:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907115" 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG { 12:25:15 INFO [http_cmd]: http status code OK 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG --- status code --- 12:25:15 DEBUG "data": { 12:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG --- http response --- 12:25:15 DEBUG "result": [ 12:25:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:25:15 DEBUG { 12:25:15 DEBUG ----- output ----- 12:25:15 DEBUG "metric": { 12:25:15 DEBUG { 12:25:15 DEBUG "pod": "ds-idrepo-0" 12:25:15 DEBUG "status": "success", 12:25:15 DEBUG }, 12:25:15 DEBUG "data": { 12:25:15 DEBUG "value": [ 12:25:15 DEBUG "resultType": "vector", 12:25:15 DEBUG 12:25:15 DEBUG "result": [ 12:25:15 DEBUG 12:25:15 DEBUG { 12:25:15 DEBUG 12:25:15 DEBUG "metric": { 12:25:15 DEBUG [...] 12:25:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:25:15 DEBUG 12:25:15 DEBUG }, 12:25:15 DEBUG 12:25:15 DEBUG "value": [ 12:25:15 DEBUG 12:25:15 DEBUG 12:25:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:25:15 DEBUG 12:25:15 DEBUG }, 12:25:15 DEBUG 12:25:15 DEBUG "value": [ 12:25:15 DEBUG [...] 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG 12:25:15 DEBUG "758558720" 12:25:15 DEBUG 12:25:15 DEBUG ] 12:25:15 DEBUG 12:25:15 DEBUG } 12:25:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:25:15 DEBUG ] 12:25:15 DEBUG }, 12:25:15 DEBUG } 12:25:15 DEBUG "value": [ 12:25:15 DEBUG } 12:25:15 DEBUG 1735907115, 12:25:15 DEBUG "14990929920" 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG ] 12:25:15 DEBUG } 12:25:15 DEBUG } 12:25:20 INFO 12:25:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:25:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:25:20 INFO [loop_until]: OK (rc = 0) 12:25:20 DEBUG --- stdout --- 12:25:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 107m 0% 3874Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 78m 0% 19052Mi 66% 12:25:20 DEBUG --- stderr --- 12:25:20 DEBUG 12:25:24 INFO 12:25:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:25:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:25:24 INFO [loop_until]: OK (rc = 0) 12:25:24 DEBUG --- stdout --- 12:25:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 6m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:25:24 DEBUG --- stderr --- 12:25:24 DEBUG 12:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907174" 12:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907174" 12:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907174" 12:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907174" 12:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907174" 12:26:14 ERROR [http_cmd]: ERROR 12:26:14 ERROR --- status code --- 12:26:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:26:14 ERROR --- http response --- 12:26:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:26:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:26:14 INFO [http_cmd]: http status code OK 12:26:14 DEBUG --- status code --- 12:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:14 DEBUG --- http response --- 12:26:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:14 INFO [http_cmd]: http status code OK 12:26:14 DEBUG --- status code --- 12:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:14 DEBUG --- http response --- 12:26:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:14 INFO [http_cmd]: http status code OK 12:26:14 DEBUG --- status code --- 12:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:14 DEBUG --- http response --- 12:26:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:14 INFO [http_cmd]: http status code OK 12:26:14 DEBUG --- status code --- 12:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:14 DEBUG --- http response --- 12:26:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907174" 12:26:14 INFO [http_cmd]: http status code OK 12:26:14 DEBUG --- status code --- 12:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:14 DEBUG --- http response --- 12:26:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907174" 12:26:14 INFO [http_cmd]: http status code OK 12:26:14 DEBUG --- status code --- 12:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:14 DEBUG --- http response --- 12:26:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907174" 12:26:14 INFO [http_cmd]: http status code OK 12:26:14 DEBUG --- status code --- 12:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:14 DEBUG --- http response --- 12:26:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735907175, "339.53192782736033" ] } ] } } 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735907175, "0" ] } ] } } 12:26:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:26:15 DEBUG ----- output ----- 12:26:15 DEBUG { 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG "status": "success", 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG "data": { 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG "resultType": "vector", 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG "result": [ 12:26:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:26:15 DEBUG { 12:26:15 DEBUG ----- output ----- 12:26:15 DEBUG "metric": { 12:26:15 DEBUG { 12:26:15 DEBUG "instance": "10.142.0.11:9100" 12:26:15 DEBUG "status": "success", 12:26:15 DEBUG }, 12:26:15 DEBUG "data": { 12:26:15 DEBUG "value": [ 12:26:15 DEBUG "resultType": "vector", 12:26:15 DEBUG 12:26:15 DEBUG "result": [ 12:26:15 DEBUG 12:26:15 DEBUG { 12:26:15 DEBUG 12:26:15 DEBUG "metric": { 12:26:15 DEBUG [...] 12:26:15 DEBUG "instance": "10.142.0.11:9100" 12:26:15 DEBUG 12:26:15 DEBUG }, 12:26:15 DEBUG 12:26:15 DEBUG "value": [ 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG "instance": "10.142.0.51:9100" 12:26:15 DEBUG 12:26:15 DEBUG }, 12:26:15 DEBUG 12:26:15 DEBUG "value": [ 12:26:15 DEBUG [...] 12:26:15 DEBUG 1735907175, 12:26:15 DEBUG 12:26:15 DEBUG "0.000666666666666534" 12:26:15 DEBUG 12:26:15 DEBUG ] 12:26:15 DEBUG 12:26:15 DEBUG } 12:26:15 DEBUG "instance": "10.142.0.51:9100" 12:26:15 DEBUG ] 12:26:15 DEBUG }, 12:26:15 DEBUG } 12:26:15 DEBUG "value": [ 12:26:15 DEBUG } 12:26:15 DEBUG 1735907175, 12:26:15 DEBUG "589445.8000000002" 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:26:15 DEBUG ----- output ----- 12:26:15 DEBUG { 12:26:15 DEBUG "status": "success", 12:26:15 DEBUG "data": { 12:26:15 DEBUG "resultType": "vector", 12:26:15 DEBUG "result": [ 12:26:15 DEBUG { 12:26:15 DEBUG "metric": { 12:26:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:26:15 DEBUG }, 12:26:15 DEBUG "value": [ 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG [...] 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG "pod": "ds-cts-0" 12:26:15 DEBUG }, 12:26:15 DEBUG "value": [ 12:26:15 DEBUG 1735907175, 12:26:15 DEBUG "0" 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:26:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:26:15 DEBUG ----- output ----- 12:26:15 DEBUG ----- output ----- 12:26:15 DEBUG { 12:26:15 DEBUG { 12:26:15 DEBUG "status": "success", 12:26:15 DEBUG "status": "success", 12:26:15 DEBUG "data": { 12:26:15 DEBUG "data": { 12:26:15 DEBUG "resultType": "vector", 12:26:15 DEBUG "resultType": "vector", 12:26:15 DEBUG "result": [ 12:26:15 DEBUG "result": [ 12:26:15 DEBUG { 12:26:15 DEBUG { 12:26:15 DEBUG "metric": { 12:26:15 DEBUG "metric": { 12:26:15 DEBUG "instance": "10.142.0.11:9100" 12:26:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:26:15 DEBUG }, 12:26:15 DEBUG }, 12:26:15 DEBUG "value": [ 12:26:15 DEBUG "value": [ 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG [...] 12:26:15 DEBUG [...] 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG "instance": "10.142.0.51:9100" 12:26:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:26:15 DEBUG }, 12:26:15 DEBUG }, 12:26:15 DEBUG "value": [ 12:26:15 DEBUG "value": [ 12:26:15 DEBUG 1735907175, 12:26:15 DEBUG 1735907175, 12:26:15 DEBUG "0.0006333333333410943" 12:26:15 DEBUG "0.033205410953933395" 12:26:15 DEBUG ] 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 DEBUG ] 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907175" 12:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907175" 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:26:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:26:15 DEBUG ----- output ----- 12:26:15 DEBUG ----- output ----- 12:26:15 DEBUG { 12:26:15 DEBUG { 12:26:15 DEBUG "status": "success", 12:26:15 DEBUG "status": "success", 12:26:15 DEBUG "data": { 12:26:15 DEBUG "data": { 12:26:15 DEBUG "resultType": "vector", 12:26:15 DEBUG "resultType": "vector", 12:26:15 DEBUG "result": [ 12:26:15 DEBUG "result": [ 12:26:15 DEBUG { 12:26:15 DEBUG { 12:26:15 DEBUG "metric": { 12:26:15 DEBUG "metric": { 12:26:15 DEBUG "instance": "10.142.0.11:9100" 12:26:15 DEBUG "instance": "10.142.0.11:9100" 12:26:15 DEBUG }, 12:26:15 DEBUG }, 12:26:15 DEBUG "value": [ 12:26:15 DEBUG "value": [ 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG [...] 12:26:15 DEBUG [...] 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG "instance": "10.142.0.51:9100" 12:26:15 DEBUG "instance": "10.142.0.51:9100" 12:26:15 DEBUG }, 12:26:15 DEBUG }, 12:26:15 DEBUG "value": [ 12:26:15 DEBUG "value": [ 12:26:15 DEBUG 1735907175, 12:26:15 DEBUG 1735907175, 12:26:15 DEBUG "28398.933333333334" 12:26:15 DEBUG "571817.7000000003" 12:26:15 DEBUG ] 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG --- http response --- 12:26:15 INFO [http_cmd]: http status code OK 12:26:15 DEBUG --- status code --- 12:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:26:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:26:15 DEBUG --- http response --- 12:26:15 DEBUG ----- output ----- 12:26:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:26:15 DEBUG { 12:26:15 DEBUG ----- output ----- 12:26:15 DEBUG "status": "success", 12:26:15 DEBUG { 12:26:15 DEBUG "data": { 12:26:15 DEBUG "status": "success", 12:26:15 DEBUG "resultType": "vector", 12:26:15 DEBUG "data": { 12:26:15 DEBUG "result": [ 12:26:15 DEBUG "resultType": "vector", 12:26:15 DEBUG { 12:26:15 DEBUG "result": [ 12:26:15 DEBUG "metric": { 12:26:15 DEBUG { 12:26:15 DEBUG "pod": "ds-idrepo-0" 12:26:15 DEBUG "metric": { 12:26:15 DEBUG }, 12:26:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:26:15 DEBUG "value": [ 12:26:15 DEBUG }, 12:26:15 DEBUG 12:26:15 DEBUG "value": [ 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG [...] 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG [...] 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG 12:26:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:26:15 DEBUG 12:26:15 DEBUG }, 12:26:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:26:15 DEBUG "value": [ 12:26:15 DEBUG }, 12:26:15 DEBUG 1735907175, 12:26:15 DEBUG "value": [ 12:26:15 DEBUG "758558720" 12:26:15 DEBUG 1735907175, 12:26:15 DEBUG ] 12:26:15 DEBUG "14987534336" 12:26:15 DEBUG } 12:26:15 DEBUG ] 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 DEBUG ] 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:15 DEBUG } 12:26:20 INFO 12:26:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:26:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:26:20 INFO [loop_until]: OK (rc = 0) 12:26:20 DEBUG --- stdout --- 12:26:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 99m 0% 3861Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 81m 0% 19053Mi 66% 12:26:20 DEBUG --- stderr --- 12:26:20 DEBUG 12:26:24 INFO 12:26:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:26:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:26:24 INFO [loop_until]: OK (rc = 0) 12:26:24 DEBUG --- stdout --- 12:26:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 495m 13478Mi rcs-645c86fdd6-j4rsk 59m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:26:24 DEBUG --- stderr --- 12:26:24 DEBUG 12:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907234" 12:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907234" 12:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907234" 12:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907234" 12:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907234" 12:27:14 INFO [http_cmd]: http status code OK 12:27:14 DEBUG --- status code --- 12:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:14 DEBUG --- http response --- 12:27:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:14 INFO [http_cmd]: http status code OK 12:27:14 INFO [http_cmd]: http status code OK 12:27:14 ERROR [http_cmd]: ERROR 12:27:14 DEBUG --- status code --- 12:27:14 DEBUG --- status code --- 12:27:14 ERROR --- status code --- 12:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:27:14 DEBUG --- http response --- 12:27:14 DEBUG --- http response --- 12:27:14 ERROR --- http response --- 12:27:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:27:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:27:14 INFO [http_cmd]: http status code OK 12:27:14 DEBUG --- status code --- 12:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:14 DEBUG --- http response --- 12:27:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907234" 12:27:14 INFO [http_cmd]: http status code OK 12:27:14 DEBUG --- status code --- 12:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:14 DEBUG --- http response --- 12:27:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907234" 12:27:14 INFO [http_cmd]: http status code OK 12:27:14 DEBUG --- status code --- 12:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:14 DEBUG --- http response --- 12:27:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907234" 12:27:14 INFO [http_cmd]: http status code OK 12:27:14 DEBUG --- status code --- 12:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:14 DEBUG --- http response --- 12:27:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG { 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG "data": { 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907235" 12:27:15 DEBUG "resultType": "vector", 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907235" 12:27:15 DEBUG "result": [ 12:27:15 DEBUG { 12:27:15 DEBUG "metric": { 12:27:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG "292.10197896238185" 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG { 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG "data": { 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG "result": [ 12:27:15 DEBUG { 12:27:15 DEBUG "metric": { 12:27:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG "0" 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG { 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG "data": { 12:27:15 DEBUG { 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG "result": [ 12:27:15 DEBUG "data": { 12:27:15 DEBUG { 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG "metric": { 12:27:15 DEBUG "result": [ 12:27:15 DEBUG "instance": "10.142.0.11:9100" 12:27:15 DEBUG { 12:27:15 DEBUG }, 12:27:15 DEBUG "metric": { 12:27:15 DEBUG "value": [ 12:27:15 DEBUG "instance": "10.142.0.11:9100" 12:27:15 DEBUG 12:27:15 DEBUG }, 12:27:15 DEBUG 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG "instance": "10.142.0.51:9100" 12:27:15 DEBUG 12:27:15 DEBUG }, 12:27:15 DEBUG 12:27:15 DEBUG "value": [ 12:27:15 DEBUG "instance": "10.142.0.51:9100" 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG }, 12:27:15 DEBUG "1255696.2666666668" 12:27:15 DEBUG "value": [ 12:27:15 DEBUG ] 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG } 12:27:15 DEBUG "0.0009999999999995642" 12:27:15 DEBUG ] 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG { 12:27:15 DEBUG "status": "success", 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907235" 12:27:15 DEBUG "data": { 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907235" 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG "result": [ 12:27:15 DEBUG { 12:27:15 DEBUG "metric": { 12:27:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG "0.0005880905687317613" 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG { 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG "data": { 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG "result": [ 12:27:15 DEBUG { 12:27:15 DEBUG "metric": { 12:27:15 DEBUG "instance": "10.142.0.11:9100" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG "instance": "10.142.0.51:9100" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG "0.001333333333342542" 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG { 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG "data": { 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG "result": [ 12:27:15 DEBUG { 12:27:15 DEBUG "metric": { 12:27:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:27:15 DEBUG }, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG "0.03004894495988048" 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907235" 12:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907235" 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG { 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG "data": { 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG { 12:27:15 DEBUG "result": [ 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG { 12:27:15 DEBUG "data": { 12:27:15 DEBUG "metric": { 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG "instance": "10.142.0.11:9100" 12:27:15 DEBUG "result": [ 12:27:15 DEBUG }, 12:27:15 DEBUG { 12:27:15 DEBUG "value": [ 12:27:15 DEBUG "metric": { 12:27:15 DEBUG 12:27:15 DEBUG "instance": "10.142.0.11:9100" 12:27:15 DEBUG 12:27:15 DEBUG }, 12:27:15 DEBUG 12:27:15 DEBUG "value": [ 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG "instance": "10.142.0.51:9100" 12:27:15 DEBUG 12:27:15 DEBUG }, 12:27:15 DEBUG 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG "instance": "10.142.0.51:9100" 12:27:15 DEBUG "25668.266666666666" 12:27:15 DEBUG }, 12:27:15 DEBUG ] 12:27:15 DEBUG "value": [ 12:27:15 DEBUG } 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG ] 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG "1240158.3" 12:27:15 DEBUG } 12:27:15 INFO [http_cmd]: http status code OK 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG --- status code --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:27:15 DEBUG } 12:27:15 DEBUG --- http response --- 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG ] 12:27:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:27:15 DEBUG { 12:27:15 DEBUG } 12:27:15 DEBUG ----- output ----- 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG } 12:27:15 DEBUG { 12:27:15 DEBUG "data": { 12:27:15 DEBUG "status": "success", 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG "data": { 12:27:15 DEBUG "result": [ 12:27:15 DEBUG "resultType": "vector", 12:27:15 DEBUG { 12:27:15 DEBUG "result": [ 12:27:15 DEBUG "metric": { 12:27:15 DEBUG { 12:27:15 DEBUG "pod": "ds-idrepo-0" 12:27:15 DEBUG "metric": { 12:27:15 DEBUG }, 12:27:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:27:15 DEBUG "value": [ 12:27:15 DEBUG }, 12:27:15 DEBUG 12:27:15 DEBUG "value": [ 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG [...] 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG 12:27:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:27:15 DEBUG 12:27:15 DEBUG }, 12:27:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:27:15 DEBUG "value": [ 12:27:15 DEBUG }, 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG "value": [ 12:27:15 DEBUG "758558720" 12:27:15 DEBUG 1735907235, 12:27:15 DEBUG ] 12:27:15 DEBUG "14991310848" 12:27:15 DEBUG } 12:27:15 DEBUG ] 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 DEBUG ] 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:15 DEBUG } 12:27:20 INFO 12:27:20 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:27:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:27:21 INFO [loop_until]: OK (rc = 0) 12:27:21 DEBUG --- stdout --- 12:27:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 105m 0% 3853Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 139m 0% 19053Mi 66% 12:27:21 DEBUG --- stderr --- 12:27:21 DEBUG 12:27:24 INFO 12:27:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:27:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:27:24 INFO [loop_until]: OK (rc = 0) 12:27:24 DEBUG --- stdout --- 12:27:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 51m 13479Mi rcs-645c86fdd6-j4rsk 40m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:27:24 DEBUG --- stderr --- 12:27:24 DEBUG 12:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907294" 12:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907294" 12:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907294" 12:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907294" 12:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907294" 12:28:14 ERROR [http_cmd]: ERROR 12:28:14 ERROR --- status code --- 12:28:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:28:14 ERROR --- http response --- 12:28:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:28:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:28:14 INFO [http_cmd]: http status code OK 12:28:14 DEBUG --- status code --- 12:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:14 DEBUG --- http response --- 12:28:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:14 INFO [http_cmd]: http status code OK 12:28:14 DEBUG --- status code --- 12:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:14 DEBUG --- http response --- 12:28:14 INFO [http_cmd]: http status code OK 12:28:14 DEBUG --- status code --- 12:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:14 DEBUG --- http response --- 12:28:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:14 INFO [http_cmd]: http status code OK 12:28:14 DEBUG --- status code --- 12:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:14 DEBUG --- http response --- 12:28:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907294" 12:28:14 INFO [http_cmd]: http status code OK 12:28:14 DEBUG --- status code --- 12:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:14 DEBUG --- http response --- 12:28:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907294" 12:28:14 INFO [http_cmd]: http status code OK 12:28:14 DEBUG --- status code --- 12:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:14 DEBUG --- http response --- 12:28:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907294" 12:28:14 INFO [http_cmd]: http status code OK 12:28:14 DEBUG --- status code --- 12:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:14 DEBUG --- http response --- 12:28:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907295" 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG { 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG "data": { 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG "result": [ 12:28:15 DEBUG { 12:28:15 DEBUG "metric": { 12:28:15 DEBUG "pod": "ds-cts-0" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG [...] 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG "703.360038879829" 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG { 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG "data": { 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG "result": [ 12:28:15 DEBUG { 12:28:15 DEBUG "metric": { 12:28:15 DEBUG "pod": "ds-cts-0" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG [...] 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG "0" 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG { 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG "data": { 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG "result": [ 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:28:15 DEBUG { 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG "metric": { 12:28:15 DEBUG { 12:28:15 DEBUG "instance": "10.142.0.11:9100" 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG }, 12:28:15 DEBUG "data": { 12:28:15 DEBUG "value": [ 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG 12:28:15 DEBUG "result": [ 12:28:15 DEBUG 12:28:15 DEBUG { 12:28:15 DEBUG 12:28:15 DEBUG "metric": { 12:28:15 DEBUG [...] 12:28:15 DEBUG "instance": "10.142.0.11:9100" 12:28:15 DEBUG 12:28:15 DEBUG }, 12:28:15 DEBUG 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG "instance": "10.142.0.51:9100" 12:28:15 DEBUG 12:28:15 DEBUG }, 12:28:15 DEBUG 12:28:15 DEBUG "value": [ 12:28:15 DEBUG [...] 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG 12:28:15 DEBUG "1241004.4333333336" 12:28:15 DEBUG 12:28:15 DEBUG ] 12:28:15 DEBUG 12:28:15 DEBUG } 12:28:15 DEBUG "instance": "10.142.0.51:9100" 12:28:15 DEBUG ] 12:28:15 DEBUG }, 12:28:15 DEBUG } 12:28:15 DEBUG "value": [ 12:28:15 DEBUG } 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG "0" 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907295" 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG { 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG "data": { 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG "result": [ 12:28:15 DEBUG { 12:28:15 DEBUG "metric": { 12:28:15 DEBUG "pod": "ds-cts-0" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG [...] 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG "0.0006713826497070235" 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG { 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG "data": { 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG "result": [ 12:28:15 DEBUG { 12:28:15 DEBUG "metric": { 12:28:15 DEBUG "instance": "10.142.0.11:9100" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG [...] 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG "instance": "10.142.0.51:9100" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG "0.0006000000000009702" 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG { 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG "data": { 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG "result": [ 12:28:15 DEBUG { 12:28:15 DEBUG "metric": { 12:28:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG [...] 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG "0.07530771022119107" 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907295" 12:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907295" 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 INFO [http_cmd]: http status code OK 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- status code --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:28:15 DEBUG --- http response --- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:28:15 DEBUG { 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG { 12:28:15 DEBUG ----- output ----- 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG { 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG { 12:28:15 DEBUG "data": { 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG "data": { 12:28:15 DEBUG "status": "success", 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG "data": { 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG "data": { 12:28:15 DEBUG "result": [ 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG "result": [ 12:28:15 DEBUG "resultType": "vector", 12:28:15 DEBUG { 12:28:15 DEBUG "result": [ 12:28:15 DEBUG { 12:28:15 DEBUG "result": [ 12:28:15 DEBUG "metric": { 12:28:15 DEBUG { 12:28:15 DEBUG "metric": { 12:28:15 DEBUG { 12:28:15 DEBUG "instance": "10.142.0.11:9100" 12:28:15 DEBUG "metric": { 12:28:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:28:15 DEBUG "metric": { 12:28:15 DEBUG }, 12:28:15 DEBUG "pod": "ds-idrepo-0" 12:28:15 DEBUG }, 12:28:15 DEBUG "instance": "10.142.0.11:9100" 12:28:15 DEBUG "value": [ 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG }, 12:28:15 DEBUG 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG [...] 12:28:15 DEBUG 12:28:15 DEBUG [...] 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG [...] 12:28:15 DEBUG 12:28:15 DEBUG [...] 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG 12:28:15 DEBUG "instance": "10.142.0.51:9100" 12:28:15 DEBUG 12:28:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:28:15 DEBUG 12:28:15 DEBUG }, 12:28:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:28:15 DEBUG }, 12:28:15 DEBUG "instance": "10.142.0.51:9100" 12:28:15 DEBUG "value": [ 12:28:15 DEBUG }, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG }, 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG "value": [ 12:28:15 DEBUG "32085.333333333332" 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG "14995607552" 12:28:15 DEBUG 1735907295, 12:28:15 DEBUG ] 12:28:15 DEBUG "758558720" 12:28:15 DEBUG ] 12:28:15 DEBUG "1224115.0666666664" 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG ] 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:15 DEBUG } 12:28:21 INFO 12:28:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:28:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:28:21 INFO [loop_until]: OK (rc = 0) 12:28:21 DEBUG --- stdout --- 12:28:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 133m 0% 3853Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 151m 0% 19052Mi 66% 12:28:21 DEBUG --- stderr --- 12:28:21 DEBUG 12:28:24 INFO 12:28:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:28:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:28:24 INFO [loop_until]: OK (rc = 0) 12:28:24 DEBUG --- stdout --- 12:28:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 98m 13478Mi rcs-645c86fdd6-j4rsk 56m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:28:24 DEBUG --- stderr --- 12:28:24 DEBUG 12:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907354" 12:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907354" 12:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907354" 12:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907354" 12:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907354" 12:29:14 INFO [http_cmd]: http status code OK 12:29:14 DEBUG --- status code --- 12:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:14 DEBUG --- http response --- 12:29:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:14 ERROR [http_cmd]: ERROR 12:29:14 ERROR --- status code --- 12:29:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:29:14 ERROR --- http response --- 12:29:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:29:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:29:14 INFO [http_cmd]: http status code OK 12:29:14 DEBUG --- status code --- 12:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:14 INFO [http_cmd]: http status code OK 12:29:14 DEBUG --- http response --- 12:29:14 INFO [http_cmd]: http status code OK 12:29:14 DEBUG --- status code --- 12:29:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:14 DEBUG --- status code --- 12:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:14 DEBUG --- http response --- 12:29:14 DEBUG --- http response --- 12:29:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907354" 12:29:14 INFO [http_cmd]: http status code OK 12:29:14 DEBUG --- status code --- 12:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:14 DEBUG --- http response --- 12:29:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907354" 12:29:14 INFO [http_cmd]: http status code OK 12:29:14 DEBUG --- status code --- 12:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:14 DEBUG --- http response --- 12:29:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907354" 12:29:14 INFO [http_cmd]: http status code OK 12:29:14 DEBUG --- status code --- 12:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:14 DEBUG --- http response --- 12:29:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 DEBUG --- status code --- 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG { 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG "data": { 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG "result": [ 12:29:15 DEBUG { 12:29:15 DEBUG { 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG "metric": { 12:29:15 DEBUG "data": { 12:29:15 DEBUG "pod": "ds-idrepo-0" 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG }, 12:29:15 DEBUG "result": [ 12:29:15 DEBUG "value": [ 12:29:15 DEBUG { 12:29:15 DEBUG 12:29:15 DEBUG "metric": { 12:29:15 DEBUG 12:29:15 DEBUG "pod": "ds-idrepo-0" 12:29:15 DEBUG 12:29:15 DEBUG }, 12:29:15 DEBUG [...] 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:29:15 DEBUG [...] 12:29:15 DEBUG }, 12:29:15 DEBUG 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG 12:29:15 DEBUG "0" 12:29:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:29:15 DEBUG ] 12:29:15 DEBUG }, 12:29:15 DEBUG } 12:29:15 DEBUG "value": [ 12:29:15 DEBUG ] 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG } 12:29:15 DEBUG "1051.1656425496612" 12:29:15 DEBUG } 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG { 12:29:15 DEBUG { 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG "data": { 12:29:15 DEBUG "data": { 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG "result": [ 12:29:15 DEBUG "result": [ 12:29:15 DEBUG { 12:29:15 DEBUG { 12:29:15 DEBUG "metric": { 12:29:15 DEBUG "metric": { 12:29:15 DEBUG "instance": "10.142.0.11:9100" 12:29:15 DEBUG "instance": "10.142.0.11:9100" 12:29:15 DEBUG }, 12:29:15 DEBUG }, 12:29:15 DEBUG "value": [ 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG [...] 12:29:15 DEBUG [...] 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG "instance": "10.142.0.51:9100" 12:29:15 DEBUG "instance": "10.142.0.51:9100" 12:29:15 DEBUG }, 12:29:15 DEBUG }, 12:29:15 DEBUG "value": [ 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG "1843508.7333333339" 12:29:15 DEBUG "0" 12:29:15 DEBUG ] 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 DEBUG ] 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG { 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG "data": { 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG "result": [ 12:29:15 DEBUG { 12:29:15 DEBUG "metric": { 12:29:15 DEBUG "pod": "ds-idrepo-0" 12:29:15 DEBUG }, 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG [...] 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:29:15 DEBUG }, 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG "0" 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG { 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG "data": { 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG "result": [ 12:29:15 DEBUG { 12:29:15 DEBUG "metric": { 12:29:15 DEBUG "instance": "10.142.0.11:9100" 12:29:15 DEBUG }, 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG [...] 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG "instance": "10.142.0.51:9100" 12:29:15 DEBUG }, 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG "0.00043333333333445503" 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG { 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG "data": { 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG "result": [ 12:29:15 DEBUG { 12:29:15 DEBUG "metric": { 12:29:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:29:15 DEBUG }, 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG [...] 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:29:15 DEBUG }, 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG "0.04951111040374092" 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907355" 12:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907355" 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG { 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG "data": { 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG "result": [ 12:29:15 DEBUG { 12:29:15 DEBUG "metric": { 12:29:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:29:15 DEBUG }, 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG [...] 12:29:15 DEBUG 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:29:15 DEBUG 12:29:15 INFO [http_cmd]: http status code OK 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG 12:29:15 DEBUG --- status code --- 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG { 12:29:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG }, 12:29:15 DEBUG --- http response --- 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:29:15 DEBUG "data": { 12:29:15 DEBUG "value": [ 12:29:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG ----- output ----- 12:29:15 DEBUG { 12:29:15 DEBUG "result": [ 12:29:15 DEBUG "14989066240" 12:29:15 DEBUG { 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG { 12:29:15 DEBUG ] 12:29:15 DEBUG "status": "success", 12:29:15 DEBUG "data": { 12:29:15 DEBUG "metric": { 12:29:15 DEBUG } 12:29:15 DEBUG "data": { 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG "instance": "10.142.0.11:9100" 12:29:15 DEBUG ] 12:29:15 DEBUG "resultType": "vector", 12:29:15 DEBUG "result": [ 12:29:15 DEBUG }, 12:29:15 DEBUG } 12:29:15 DEBUG "result": [ 12:29:15 DEBUG { 12:29:15 DEBUG "value": [ 12:29:15 DEBUG } 12:29:15 DEBUG { 12:29:15 DEBUG "metric": { 12:29:15 DEBUG 12:29:15 DEBUG "metric": { 12:29:15 DEBUG "instance": "10.142.0.11:9100" 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG "pod": "ds-idrepo-0" 12:29:15 DEBUG [...] 12:29:15 DEBUG }, 12:29:15 DEBUG }, 12:29:15 DEBUG 12:29:15 DEBUG "value": [ 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG "instance": "10.142.0.51:9100" 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG }, 12:29:15 DEBUG [...] 12:29:15 DEBUG [...] 12:29:15 DEBUG "value": [ 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG "25531.733333333334" 12:29:15 DEBUG 12:29:15 DEBUG 12:29:15 DEBUG ] 12:29:15 DEBUG "instance": "10.142.0.51:9100" 12:29:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:29:15 DEBUG } 12:29:15 DEBUG }, 12:29:15 DEBUG }, 12:29:15 DEBUG ] 12:29:15 DEBUG "value": [ 12:29:15 DEBUG "value": [ 12:29:15 DEBUG } 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG 1735907355, 12:29:15 DEBUG } 12:29:15 DEBUG "1826616.633333333" 12:29:15 DEBUG "758558720" 12:29:15 DEBUG ] 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 DEBUG ] 12:29:15 DEBUG ] 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:15 DEBUG } 12:29:21 INFO 12:29:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:29:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:29:21 INFO [loop_until]: OK (rc = 0) 12:29:21 DEBUG --- stdout --- 12:29:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 200m 1% 3863Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 176m 0% 19053Mi 66% 12:29:21 DEBUG --- stderr --- 12:29:21 DEBUG 12:29:24 INFO 12:29:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:29:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:29:24 INFO [loop_until]: OK (rc = 0) 12:29:24 DEBUG --- stdout --- 12:29:24 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 107m 13479Mi rcs-645c86fdd6-j4rsk 66m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:29:24 DEBUG --- stderr --- 12:29:24 DEBUG 12:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907414" 12:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907414" 12:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907414" 12:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907414" 12:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907414" 12:30:14 INFO [http_cmd]: http status code OK 12:30:14 ERROR [http_cmd]: ERROR 12:30:14 DEBUG --- status code --- 12:30:14 ERROR --- status code --- 12:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:30:14 DEBUG --- http response --- 12:30:14 ERROR --- http response --- 12:30:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:30:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:30:14 INFO [http_cmd]: http status code OK 12:30:14 DEBUG --- status code --- 12:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:14 DEBUG --- http response --- 12:30:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:14 INFO [http_cmd]: http status code OK 12:30:14 DEBUG --- status code --- 12:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:14 DEBUG --- http response --- 12:30:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:14 INFO [http_cmd]: http status code OK 12:30:14 DEBUG --- status code --- 12:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:14 DEBUG --- http response --- 12:30:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907414" 12:30:14 INFO [http_cmd]: http status code OK 12:30:14 DEBUG --- status code --- 12:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:14 DEBUG --- http response --- 12:30:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907414" 12:30:14 INFO [http_cmd]: http status code OK 12:30:14 DEBUG --- status code --- 12:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:14 DEBUG --- http response --- 12:30:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907414" 12:30:14 INFO [http_cmd]: http status code OK 12:30:14 DEBUG --- status code --- 12:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:14 DEBUG --- http response --- 12:30:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907415" 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG { 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG "data": { 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG "result": [ 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG { 12:30:15 DEBUG { 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG "instance": "10.142.0.11:9100" 12:30:15 DEBUG "data": { 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:30:15 DEBUG }, 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG "value": [ 12:30:15 DEBUG "result": [ 12:30:15 DEBUG { 12:30:15 DEBUG 12:30:15 DEBUG { 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "data": { 12:30:15 DEBUG 12:30:15 DEBUG "pod": "ds-idrepo-0" 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG [...] 12:30:15 DEBUG }, 12:30:15 DEBUG "result": [ 12:30:15 DEBUG 12:30:15 DEBUG "value": [ 12:30:15 DEBUG { 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "metric": { 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "instance": "10.142.0.11:9100" 12:30:15 DEBUG "instance": "10.142.0.51:9100" 12:30:15 DEBUG 12:30:15 DEBUG }, 12:30:15 DEBUG }, 12:30:15 DEBUG [...] 12:30:15 DEBUG "value": [ 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "0.00033333333333492493" 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG ] 12:30:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:30:15 DEBUG [...] 12:30:15 DEBUG } 12:30:15 DEBUG }, 12:30:15 DEBUG 12:30:15 DEBUG ] 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 12:30:15 DEBUG } 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG 12:30:15 DEBUG } 12:30:15 DEBUG "470.8345361415009" 12:30:15 DEBUG "instance": "10.142.0.51:9100" 12:30:15 DEBUG ] 12:30:15 DEBUG }, 12:30:15 DEBUG } 12:30:15 DEBUG "value": [ 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG } 12:30:15 DEBUG "2388588.266666667" 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG { 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG "data": { 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG "result": [ 12:30:15 DEBUG { 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "pod": "ds-idrepo-0" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG [...] 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG "0" 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907415" 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG { 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG "data": { 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG "result": [ 12:30:15 DEBUG { 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "pod": "ds-idrepo-0" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG [...] 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG "0.0008984037563923486" 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG { 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG "data": { 12:30:15 DEBUG { 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG "result": [ 12:30:15 DEBUG "data": { 12:30:15 DEBUG { 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "result": [ 12:30:15 DEBUG "instance": "10.142.0.11:9100" 12:30:15 DEBUG { 12:30:15 DEBUG }, 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "value": [ 12:30:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:30:15 DEBUG 12:30:15 DEBUG }, 12:30:15 DEBUG 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG [...] 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG [...] 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "instance": "10.142.0.51:9100" 12:30:15 DEBUG 12:30:15 DEBUG }, 12:30:15 DEBUG 12:30:15 DEBUG "value": [ 12:30:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG }, 12:30:15 DEBUG "0.0004333333333354024" 12:30:15 DEBUG "value": [ 12:30:15 DEBUG ] 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG } 12:30:15 DEBUG "0.07293845880498191" 12:30:15 DEBUG ] 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907415" 12:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907415" 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG { 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG "data": { 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG "result": [ 12:30:15 DEBUG { 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG [...] 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG "14991134720" 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG { 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG "data": { 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG "result": [ 12:30:15 DEBUG { 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "instance": "10.142.0.11:9100" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG [...] 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "instance": "10.142.0.51:9100" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG "2370546" 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG { 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG "data": { 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG "result": [ 12:30:15 DEBUG { 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "instance": "10.142.0.11:9100" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG [...] 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "instance": "10.142.0.51:9100" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG "25941.333333333332" 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:15 INFO [http_cmd]: http status code OK 12:30:15 DEBUG --- status code --- 12:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:30:15 DEBUG --- http response --- 12:30:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:30:15 DEBUG ----- output ----- 12:30:15 DEBUG { 12:30:15 DEBUG "status": "success", 12:30:15 DEBUG "data": { 12:30:15 DEBUG "resultType": "vector", 12:30:15 DEBUG "result": [ 12:30:15 DEBUG { 12:30:15 DEBUG "metric": { 12:30:15 DEBUG "pod": "ds-idrepo-0" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG [...] 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG 12:30:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:30:15 DEBUG }, 12:30:15 DEBUG "value": [ 12:30:15 DEBUG 1735907415, 12:30:15 DEBUG "758558720" 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG ] 12:30:15 DEBUG } 12:30:15 DEBUG } 12:30:21 INFO 12:30:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:30:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:30:21 INFO [loop_until]: OK (rc = 0) 12:30:21 DEBUG --- stdout --- 12:30:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 227m 1% 3864Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 226m 0% 19052Mi 66% 12:30:21 DEBUG --- stderr --- 12:30:21 DEBUG 12:30:24 INFO 12:30:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:30:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:30:25 INFO [loop_until]: OK (rc = 0) 12:30:25 DEBUG --- stdout --- 12:30:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 154m 13479Mi rcs-645c86fdd6-j4rsk 100m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:30:25 DEBUG --- stderr --- 12:30:25 DEBUG 12:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907474" 12:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907474" 12:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907474" 12:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907474" 12:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907474" 12:31:14 ERROR [http_cmd]: ERROR 12:31:14 ERROR --- status code --- 12:31:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:31:14 ERROR --- http response --- 12:31:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:31:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:31:14 INFO [http_cmd]: http status code OK 12:31:14 DEBUG --- status code --- 12:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:14 INFO [http_cmd]: http status code OK 12:31:14 DEBUG --- http response --- 12:31:14 DEBUG --- status code --- 12:31:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:14 DEBUG --- http response --- 12:31:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:14 INFO [http_cmd]: http status code OK 12:31:14 DEBUG --- status code --- 12:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:14 DEBUG --- http response --- 12:31:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:14 INFO [http_cmd]: http status code OK 12:31:14 DEBUG --- status code --- 12:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:14 DEBUG --- http response --- 12:31:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907474" 12:31:14 INFO [http_cmd]: http status code OK 12:31:14 DEBUG --- status code --- 12:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:14 DEBUG --- http response --- 12:31:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907474" 12:31:14 INFO [http_cmd]: http status code OK 12:31:14 DEBUG --- status code --- 12:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:14 DEBUG --- http response --- 12:31:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907474" 12:31:14 INFO [http_cmd]: http status code OK 12:31:14 DEBUG --- status code --- 12:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:14 DEBUG --- http response --- 12:31:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG { 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG "data": { 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG "result": [ 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:31:15 DEBUG { 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG "metric": { 12:31:15 DEBUG { 12:31:15 DEBUG "pod": "ds-cts-0" 12:31:15 DEBUG "status": "success", 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG }, 12:31:15 DEBUG "data": { 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG "value": [ 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG 12:31:15 DEBUG "result": [ 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG 12:31:15 DEBUG { 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:31:15 DEBUG 12:31:15 DEBUG "metric": { 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG [...] 12:31:15 DEBUG "instance": "10.142.0.11:9100" 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG { 12:31:15 DEBUG 12:31:15 DEBUG }, 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG "value": [ 12:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:31:15 DEBUG "data": { 12:31:15 DEBUG 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG 12:31:15 DEBUG "result": [ 12:31:15 DEBUG }, 12:31:15 DEBUG --- http response --- 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907475" 12:31:15 DEBUG 12:31:15 DEBUG { 12:31:15 DEBUG "value": [ 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:31:15 DEBUG [...] 12:31:15 DEBUG "metric": { 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG 12:31:15 DEBUG "instance": "10.142.0.11:9100" 12:31:15 DEBUG "450.4233715772883" 12:31:15 DEBUG { 12:31:15 DEBUG 12:31:15 DEBUG }, 12:31:15 DEBUG ] 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG 12:31:15 DEBUG "value": [ 12:31:15 DEBUG } 12:31:15 DEBUG "data": { 12:31:15 DEBUG "instance": "10.142.0.51:9100" 12:31:15 DEBUG 12:31:15 DEBUG ] 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG }, 12:31:15 DEBUG 12:31:15 DEBUG } 12:31:15 DEBUG "result": [ 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 12:31:15 DEBUG } 12:31:15 DEBUG { 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG [...] 12:31:15 DEBUG "metric": { 12:31:15 DEBUG "1735342.8666666667" 12:31:15 DEBUG 12:31:15 DEBUG "pod": "ds-cts-0" 12:31:15 DEBUG ] 12:31:15 DEBUG 12:31:15 DEBUG }, 12:31:15 DEBUG } 12:31:15 DEBUG 12:31:15 DEBUG "value": [ 12:31:15 DEBUG ] 12:31:15 DEBUG "instance": "10.142.0.51:9100" 12:31:15 DEBUG 12:31:15 DEBUG } 12:31:15 DEBUG }, 12:31:15 DEBUG 12:31:15 DEBUG } 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG [...] 12:31:15 DEBUG "0" 12:31:15 DEBUG 12:31:15 DEBUG ] 12:31:15 DEBUG 12:31:15 DEBUG } 12:31:15 DEBUG 12:31:15 DEBUG ] 12:31:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:31:15 DEBUG } 12:31:15 DEBUG }, 12:31:15 DEBUG } 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG "0" 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG { 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG "data": { 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG "result": [ 12:31:15 DEBUG { 12:31:15 DEBUG "metric": { 12:31:15 DEBUG "pod": "ds-cts-0" 12:31:15 DEBUG }, 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG [...] 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:31:15 DEBUG }, 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG "0.0005249807558371336" 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG } 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG { 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG "data": { 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG "result": [ 12:31:15 DEBUG { 12:31:15 DEBUG "metric": { 12:31:15 DEBUG "instance": "10.142.0.11:9100" 12:31:15 DEBUG }, 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG [...] 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG "instance": "10.142.0.51:9100" 12:31:15 DEBUG }, 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG "0.00040000000000190995" 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG } 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG { 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG "data": { 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG "result": [ 12:31:15 DEBUG { 12:31:15 DEBUG "metric": { 12:31:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:31:15 DEBUG }, 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG [...] 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:31:15 DEBUG }, 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG "0.14527382694187338" 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG } 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907475" 12:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907475" 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG { 12:31:15 DEBUG { 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG "data": { 12:31:15 DEBUG "data": { 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG "result": [ 12:31:15 DEBUG "result": [ 12:31:15 DEBUG { 12:31:15 DEBUG { 12:31:15 DEBUG "metric": { 12:31:15 DEBUG "metric": { 12:31:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:31:15 DEBUG "instance": "10.142.0.11:9100" 12:31:15 DEBUG }, 12:31:15 DEBUG }, 12:31:15 DEBUG "value": [ 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG [...] 12:31:15 DEBUG [...] 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG 12:31:15 INFO [http_cmd]: http status code OK 12:31:15 DEBUG 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG --- status code --- 12:31:15 DEBUG 12:31:15 DEBUG }, 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:31:15 DEBUG 12:31:15 DEBUG "value": [ 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:31:15 DEBUG --- http response --- 12:31:15 DEBUG "instance": "10.142.0.51:9100" 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:31:15 DEBUG }, 12:31:15 DEBUG "14988832768" 12:31:15 DEBUG { 12:31:15 DEBUG ----- output ----- 12:31:15 DEBUG "value": [ 12:31:15 DEBUG ] 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG "data": { 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG } 12:31:15 DEBUG { 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG "result": [ 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG } 12:31:15 DEBUG { 12:31:15 DEBUG "status": "success", 12:31:15 DEBUG "1717848" 12:31:15 DEBUG "metric": { 12:31:15 DEBUG "data": { 12:31:15 DEBUG ] 12:31:15 DEBUG "instance": "10.142.0.11:9100" 12:31:15 DEBUG "resultType": "vector", 12:31:15 DEBUG } 12:31:15 DEBUG }, 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 12:31:15 DEBUG "result": [ 12:31:15 DEBUG ] 12:31:15 DEBUG 12:31:15 DEBUG { 12:31:15 DEBUG } 12:31:15 DEBUG 12:31:15 DEBUG "metric": { 12:31:15 DEBUG } 12:31:15 DEBUG [...] 12:31:15 DEBUG "pod": "ds-idrepo-0" 12:31:15 DEBUG 12:31:15 DEBUG }, 12:31:15 DEBUG 12:31:15 DEBUG "value": [ 12:31:15 DEBUG 12:31:15 DEBUG 12:31:15 DEBUG "instance": "10.142.0.51:9100" 12:31:15 DEBUG 12:31:15 DEBUG }, 12:31:15 DEBUG 12:31:15 DEBUG "value": [ 12:31:15 DEBUG [...] 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG 12:31:15 DEBUG "25122.13333333333" 12:31:15 DEBUG 12:31:15 DEBUG ] 12:31:15 DEBUG 12:31:15 DEBUG } 12:31:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:31:15 DEBUG ] 12:31:15 DEBUG }, 12:31:15 DEBUG } 12:31:15 DEBUG "value": [ 12:31:15 DEBUG } 12:31:15 DEBUG 1735907475, 12:31:15 DEBUG "758558720" 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG ] 12:31:15 DEBUG } 12:31:15 DEBUG } 12:31:21 INFO 12:31:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:31:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:31:21 INFO [loop_until]: OK (rc = 0) 12:31:21 DEBUG --- stdout --- 12:31:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 192m 1% 3854Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 168m 0% 19051Mi 66% 12:31:21 DEBUG --- stderr --- 12:31:21 DEBUG 12:31:25 INFO 12:31:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:31:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:31:25 INFO [loop_until]: OK (rc = 0) 12:31:25 DEBUG --- stdout --- 12:31:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 72m 13478Mi rcs-645c86fdd6-j4rsk 126m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:31:25 DEBUG --- stderr --- 12:31:25 DEBUG 12:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907534" 12:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907534" 12:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907534" 12:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907534" 12:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907534" 12:32:14 ERROR [http_cmd]: ERROR 12:32:14 ERROR --- status code --- 12:32:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:32:14 ERROR --- http response --- 12:32:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:32:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:32:14 INFO [http_cmd]: http status code OK 12:32:14 DEBUG --- status code --- 12:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:14 DEBUG --- http response --- 12:32:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:14 INFO [http_cmd]: http status code OK 12:32:14 DEBUG --- status code --- 12:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:14 DEBUG --- http response --- 12:32:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:14 INFO [http_cmd]: http status code OK 12:32:14 DEBUG --- status code --- 12:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:14 DEBUG --- http response --- 12:32:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:14 INFO [http_cmd]: http status code OK 12:32:14 DEBUG --- status code --- 12:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:14 DEBUG --- http response --- 12:32:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907534" 12:32:14 INFO [http_cmd]: http status code OK 12:32:14 DEBUG --- status code --- 12:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:14 DEBUG --- http response --- 12:32:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907534" 12:32:14 INFO [http_cmd]: http status code OK 12:32:14 DEBUG --- status code --- 12:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:14 DEBUG --- http response --- 12:32:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907534" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG { 12:32:15 DEBUG "status": "success", 12:32:15 DEBUG "data": { 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG "result": [ 12:32:15 DEBUG { 12:32:15 DEBUG "metric": { 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG }, 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG { 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG "value": [ 12:32:15 DEBUG { 12:32:15 DEBUG "status": "success", 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG 12:32:15 DEBUG "status": "success", 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907535" 12:32:15 DEBUG "data": { 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 DEBUG 12:32:15 DEBUG "data": { 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG "result": [ 12:32:15 DEBUG { 12:32:15 DEBUG [...] 12:32:15 DEBUG "result": [ 12:32:15 DEBUG { 12:32:15 DEBUG "status": "success", 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907535" 12:32:15 DEBUG 12:32:15 DEBUG { 12:32:15 DEBUG "metric": { 12:32:15 DEBUG "data": { 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG 12:32:15 DEBUG "metric": { 12:32:15 DEBUG "instance": "10.142.0.11:9100" 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG 12:32:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:32:15 DEBUG }, 12:32:15 DEBUG "result": [ 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:32:15 DEBUG }, 12:32:15 DEBUG "value": [ 12:32:15 DEBUG { 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 DEBUG }, 12:32:15 DEBUG "value": [ 12:32:15 DEBUG 12:32:15 DEBUG "metric": { 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG "value": [ 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG "instance": "10.142.0.11:9100" 12:32:15 DEBUG { 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG "0" 12:32:15 DEBUG 12:32:15 DEBUG }, 12:32:15 DEBUG "status": "success", 12:32:15 DEBUG 12:32:15 DEBUG ] 12:32:15 DEBUG [...] 12:32:15 DEBUG "value": [ 12:32:15 DEBUG "data": { 12:32:15 DEBUG 12:32:15 DEBUG } 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG [...] 12:32:15 DEBUG ] 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG "result": [ 12:32:15 DEBUG 12:32:15 DEBUG } 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG { 12:32:15 DEBUG 12:32:15 DEBUG } 12:32:15 DEBUG "instance": "10.142.0.51:9100" 12:32:15 DEBUG [...] 12:32:15 DEBUG "metric": { 12:32:15 DEBUG 12:32:15 DEBUG }, 12:32:15 DEBUG 12:32:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:32:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:32:15 DEBUG "value": [ 12:32:15 DEBUG 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG }, 12:32:15 DEBUG }, 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG "value": [ 12:32:15 DEBUG "value": [ 12:32:15 DEBUG "0.001000000000000038" 12:32:15 DEBUG "instance": "10.142.0.51:9100" 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG ] 12:32:15 DEBUG }, 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG 12:32:15 DEBUG "312.3990178884602" 12:32:15 DEBUG } 12:32:15 DEBUG "value": [ 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 DEBUG 12:32:15 DEBUG ] 12:32:15 DEBUG ] 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG [...] 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:15 DEBUG "1241644.3000000003" 12:32:15 DEBUG { 12:32:15 DEBUG 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG ] 12:32:15 DEBUG } 12:32:15 DEBUG ] 12:32:15 DEBUG } 12:32:15 DEBUG ] 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:15 DEBUG "status": "success", 12:32:15 DEBUG 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG } 12:32:15 DEBUG "data": { 12:32:15 DEBUG 12:32:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:32:15 DEBUG }, 12:32:15 DEBUG "value": [ 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG } 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 DEBUG "result": [ 12:32:15 DEBUG "0.0003413813486497151" 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG { 12:32:15 DEBUG ] 12:32:15 DEBUG { 12:32:15 DEBUG "metric": { 12:32:15 DEBUG } 12:32:15 DEBUG "status": "success", 12:32:15 DEBUG "instance": "10.142.0.11:9100" 12:32:15 DEBUG ] 12:32:15 DEBUG "data": { 12:32:15 DEBUG }, 12:32:15 DEBUG } 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG "value": [ 12:32:15 DEBUG } 12:32:15 DEBUG "result": [ 12:32:15 DEBUG 12:32:15 DEBUG { 12:32:15 DEBUG 12:32:15 DEBUG "metric": { 12:32:15 DEBUG 12:32:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:32:15 DEBUG [...] 12:32:15 DEBUG }, 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG "value": [ 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG "instance": "10.142.0.51:9100" 12:32:15 DEBUG 12:32:15 DEBUG }, 12:32:15 DEBUG 12:32:15 DEBUG "value": [ 12:32:15 DEBUG [...] 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG 12:32:15 DEBUG "0.0012333333333344853" 12:32:15 DEBUG 12:32:15 DEBUG ] 12:32:15 DEBUG 12:32:15 DEBUG } 12:32:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:32:15 DEBUG ] 12:32:15 DEBUG }, 12:32:15 DEBUG } 12:32:15 DEBUG "value": [ 12:32:15 DEBUG } 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG "0.13047971605063935" 12:32:15 DEBUG ] 12:32:15 DEBUG } 12:32:15 DEBUG ] 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907535" 12:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907535" 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG { 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 DEBUG "status": "success", 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG "data": { 12:32:15 DEBUG { 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG "status": "success", 12:32:15 DEBUG "result": [ 12:32:15 DEBUG "data": { 12:32:15 DEBUG { 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG "metric": { 12:32:15 DEBUG "result": [ 12:32:15 DEBUG "instance": "10.142.0.11:9100" 12:32:15 DEBUG { 12:32:15 DEBUG }, 12:32:15 DEBUG "metric": { 12:32:15 DEBUG "value": [ 12:32:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:32:15 DEBUG 12:32:15 DEBUG }, 12:32:15 DEBUG 12:32:15 DEBUG "value": [ 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG [...] 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG [...] 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG "instance": "10.142.0.51:9100" 12:32:15 DEBUG 12:32:15 DEBUG }, 12:32:15 DEBUG 12:32:15 DEBUG "value": [ 12:32:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG }, 12:32:15 DEBUG "1224796.8999999997" 12:32:15 DEBUG "value": [ 12:32:15 DEBUG ] 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG } 12:32:15 DEBUG "14990049280" 12:32:15 DEBUG ] 12:32:15 DEBUG ] 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:15 DEBUG ] 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 INFO [http_cmd]: http status code OK 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG --- status code --- 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG --- http response --- 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:32:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG ----- output ----- 12:32:15 DEBUG { 12:32:15 DEBUG { 12:32:15 DEBUG "status": "success", 12:32:15 DEBUG "status": "success", 12:32:15 DEBUG "data": { 12:32:15 DEBUG "data": { 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG "resultType": "vector", 12:32:15 DEBUG "result": [ 12:32:15 DEBUG "result": [ 12:32:15 DEBUG { 12:32:15 DEBUG { 12:32:15 DEBUG "metric": { 12:32:15 DEBUG "metric": { 12:32:15 DEBUG "pod": "ds-idrepo-0" 12:32:15 DEBUG "instance": "10.142.0.11:9100" 12:32:15 DEBUG }, 12:32:15 DEBUG }, 12:32:15 DEBUG "value": [ 12:32:15 DEBUG "value": [ 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG [...] 12:32:15 DEBUG [...] 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG 12:32:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:32:15 DEBUG "instance": "10.142.0.51:9100" 12:32:15 DEBUG }, 12:32:15 DEBUG }, 12:32:15 DEBUG "value": [ 12:32:15 DEBUG "value": [ 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG 1735907535, 12:32:15 DEBUG "758558720" 12:32:15 DEBUG "26214.4" 12:32:15 DEBUG ] 12:32:15 DEBUG ] 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:15 DEBUG ] 12:32:15 DEBUG ] 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:15 DEBUG } 12:32:21 INFO 12:32:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:32:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:32:21 INFO [loop_until]: OK (rc = 0) 12:32:21 DEBUG --- stdout --- 12:32:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 173m 1% 3853Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 744m 2% 19057Mi 67% 12:32:21 DEBUG --- stderr --- 12:32:21 DEBUG 12:32:25 INFO 12:32:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:32:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:32:25 INFO [loop_until]: OK (rc = 0) 12:32:25 DEBUG --- stdout --- 12:32:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 490m 13479Mi rcs-645c86fdd6-j4rsk 65m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:32:25 DEBUG --- stderr --- 12:32:25 DEBUG 12:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907594" 12:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907594" 12:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907594" 12:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907594" 12:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907594" 12:33:14 INFO [http_cmd]: http status code OK 12:33:14 DEBUG --- status code --- 12:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:14 DEBUG --- http response --- 12:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:14 INFO [http_cmd]: http status code OK 12:33:14 DEBUG --- status code --- 12:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:14 DEBUG --- http response --- 12:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:14 INFO [http_cmd]: http status code OK 12:33:14 DEBUG --- status code --- 12:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:14 DEBUG --- http response --- 12:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:14 INFO [http_cmd]: http status code OK 12:33:14 DEBUG --- status code --- 12:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:14 DEBUG --- http response --- 12:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:14 ERROR [http_cmd]: ERROR 12:33:14 ERROR --- status code --- 12:33:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:33:14 ERROR --- http response --- 12:33:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:33:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907594" 12:33:14 INFO [http_cmd]: http status code OK 12:33:14 DEBUG --- status code --- 12:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:14 DEBUG --- http response --- 12:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907594" 12:33:14 INFO [http_cmd]: http status code OK 12:33:14 DEBUG --- status code --- 12:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:14 DEBUG --- http response --- 12:33:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG { 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG "data": { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG "result": [ 12:33:15 DEBUG { 12:33:15 DEBUG "metric": { 12:33:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG [...] 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG "pod": "ds-cts-0" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG "0" 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG { 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG "data": { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG "result": [ 12:33:15 DEBUG { 12:33:15 DEBUG "metric": { 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG }, 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG "value": [ 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:33:15 DEBUG 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG { 12:33:15 DEBUG { 12:33:15 DEBUG [...] 12:33:15 DEBUG { 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG "data": { 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG "data": { 12:33:15 DEBUG 12:33:15 DEBUG "data": { 12:33:15 DEBUG { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG "result": [ 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG "result": [ 12:33:15 DEBUG "pod": "ds-cts-0" 12:33:15 DEBUG "result": [ 12:33:15 DEBUG "data": { 12:33:15 DEBUG { 12:33:15 DEBUG "data": { 12:33:15 DEBUG { 12:33:15 DEBUG }, 12:33:15 DEBUG { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG "metric": { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG "metric": { 12:33:15 DEBUG "value": [ 12:33:15 DEBUG "metric": { 12:33:15 DEBUG "result": [ 12:33:15 DEBUG "instance": "10.142.0.11:9100" 12:33:15 DEBUG "result": [ 12:33:15 DEBUG "instance": "10.142.0.11:9100" 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG "instance": "10.142.0.11:9100" 12:33:15 DEBUG }, 12:33:15 DEBUG }, 12:33:15 DEBUG { 12:33:15 DEBUG }, 12:33:15 DEBUG "688.0795591990324" 12:33:15 DEBUG { 12:33:15 DEBUG "value": [ 12:33:15 DEBUG "value": [ 12:33:15 DEBUG "metric": { 12:33:15 DEBUG "value": [ 12:33:15 DEBUG ] 12:33:15 DEBUG "metric": { 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG }, 12:33:15 DEBUG } 12:33:15 DEBUG 12:33:15 DEBUG }, 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG "value": [ 12:33:15 DEBUG ] 12:33:15 DEBUG [...] 12:33:15 DEBUG "value": [ 12:33:15 DEBUG [...] 12:33:15 DEBUG [...] 12:33:15 DEBUG 12:33:15 DEBUG } 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG } 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG [...] 12:33:15 DEBUG "instance": "10.142.0.51:9100" 12:33:15 DEBUG [...] 12:33:15 DEBUG "instance": "10.142.0.51:9100" 12:33:15 DEBUG "instance": "10.142.0.51:9100" 12:33:15 DEBUG 12:33:15 DEBUG }, 12:33:15 DEBUG 12:33:15 DEBUG }, 12:33:15 DEBUG }, 12:33:15 DEBUG 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 12:33:15 DEBUG "value": [ 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG "pod": "ds-cts-0" 12:33:15 DEBUG "1241552.4000000001" 12:33:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:33:15 DEBUG "0.0003333333333330302" 12:33:15 DEBUG "0.0005333333333283008" 12:33:15 DEBUG }, 12:33:15 DEBUG ] 12:33:15 DEBUG }, 12:33:15 DEBUG ] 12:33:15 DEBUG ] 12:33:15 DEBUG "value": [ 12:33:15 DEBUG } 12:33:15 DEBUG "value": [ 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG ] 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG ] 12:33:15 DEBUG ] 12:33:15 DEBUG "0.0026824687542069273" 12:33:15 DEBUG } 12:33:15 DEBUG "0.8797337241530593" 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 DEBUG ] 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907595" 12:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907595" 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG { 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG "data": { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG "result": [ 12:33:15 DEBUG { 12:33:15 DEBUG "metric": { 12:33:15 DEBUG "instance": "10.142.0.11:9100" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG [...] 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG "instance": "10.142.0.51:9100" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG "1224656.4333333333" 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG { 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG "data": { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG "result": [ 12:33:15 DEBUG { 12:33:15 DEBUG "metric": { 12:33:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG [...] 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG "14998859776" 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG { 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG "data": { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG "result": [ 12:33:15 DEBUG { 12:33:15 DEBUG "metric": { 12:33:15 DEBUG "pod": "ds-idrepo-0" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG [...] 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG "758558720" 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:15 INFO [http_cmd]: http status code OK 12:33:15 DEBUG --- status code --- 12:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:33:15 DEBUG --- http response --- 12:33:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:33:15 DEBUG ----- output ----- 12:33:15 DEBUG { 12:33:15 DEBUG "status": "success", 12:33:15 DEBUG "data": { 12:33:15 DEBUG "resultType": "vector", 12:33:15 DEBUG "result": [ 12:33:15 DEBUG { 12:33:15 DEBUG "metric": { 12:33:15 DEBUG "instance": "10.142.0.11:9100" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG [...] 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG 12:33:15 DEBUG "instance": "10.142.0.51:9100" 12:33:15 DEBUG }, 12:33:15 DEBUG "value": [ 12:33:15 DEBUG 1735907595, 12:33:15 DEBUG "32631.466666666667" 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG ] 12:33:15 DEBUG } 12:33:15 DEBUG } 12:33:21 INFO 12:33:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:33:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:33:21 INFO [loop_until]: OK (rc = 0) 12:33:21 DEBUG --- stdout --- 12:33:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 166m 1% 3856Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 142m 0% 19056Mi 67% 12:33:21 DEBUG --- stderr --- 12:33:21 DEBUG 12:33:25 INFO 12:33:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:33:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:33:25 INFO [loop_until]: OK (rc = 0) 12:33:25 DEBUG --- stdout --- 12:33:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 60m 13478Mi rcs-645c86fdd6-j4rsk 60m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:33:25 DEBUG --- stderr --- 12:33:25 DEBUG 12:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907654" 12:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907654" 12:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907654" 12:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907654" 12:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907654" 12:34:14 INFO [http_cmd]: http status code OK 12:34:14 INFO [http_cmd]: http status code OK 12:34:14 ERROR [http_cmd]: ERROR 12:34:14 INFO [http_cmd]: http status code OK 12:34:14 INFO [http_cmd]: http status code OK 12:34:14 DEBUG --- status code --- 12:34:14 DEBUG --- status code --- 12:34:14 ERROR --- status code --- 12:34:14 DEBUG --- status code --- 12:34:14 DEBUG --- status code --- 12:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:14 DEBUG --- http response --- 12:34:14 DEBUG --- http response --- 12:34:14 ERROR --- http response --- 12:34:14 DEBUG --- http response --- 12:34:14 DEBUG --- http response --- 12:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907654" 12:34:14 INFO [http_cmd]: http status code OK 12:34:14 DEBUG --- status code --- 12:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:14 DEBUG --- http response --- 12:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907654" 12:34:14 INFO [http_cmd]: http status code OK 12:34:14 DEBUG --- status code --- 12:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:14 DEBUG --- http response --- 12:34:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- http response --- 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907655" 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:34:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:34:15 DEBUG ----- output ----- 12:34:15 DEBUG ----- output ----- 12:34:15 DEBUG { 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG { 12:34:15 DEBUG "status": "success", 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG "status": "success", 12:34:15 DEBUG "data": { 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG "data": { 12:34:15 DEBUG "resultType": "vector", 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG "resultType": "vector", 12:34:15 DEBUG "result": [ 12:34:15 DEBUG "result": [ 12:34:15 DEBUG { 12:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-j4rsk" }, "value": [ 1735907655, "0" ] } ] } } 12:34:15 DEBUG { 12:34:15 DEBUG "metric": { 12:34:15 DEBUG "metric": { 12:34:15 DEBUG "instance": "10.142.0.11:9100" 12:34:15 DEBUG "instance": "10.142.0.11:9100" 12:34:15 DEBUG }, 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG }, 12:34:15 DEBUG "value": [ 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG "value": [ 12:34:15 DEBUG 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:34:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:34:15 DEBUG 12:34:15 DEBUG [...] 12:34:15 DEBUG ----- output ----- 12:34:15 DEBUG ----- output ----- 12:34:15 DEBUG [...] 12:34:15 DEBUG 12:34:15 DEBUG { 12:34:15 DEBUG { 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG "status": "success", 12:34:15 DEBUG "status": "success", 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG "data": { 12:34:15 DEBUG "data": { 12:34:15 DEBUG 12:34:15 DEBUG "instance": "10.142.0.51:9100" 12:34:15 DEBUG "resultType": "vector", 12:34:15 DEBUG "resultType": "vector", 12:34:15 DEBUG "instance": "10.142.0.51:9100" 12:34:15 DEBUG }, 12:34:15 DEBUG "result": [ 12:34:15 DEBUG "result": [ 12:34:15 DEBUG }, 12:34:15 DEBUG "value": [ 12:34:15 DEBUG { 12:34:15 DEBUG { 12:34:15 DEBUG "value": [ 12:34:15 DEBUG 1735907655, 12:34:15 DEBUG "metric": { 12:34:15 DEBUG "metric": { 12:34:15 DEBUG 1735907655, 12:34:15 DEBUG "975008.8666666668" 12:34:15 DEBUG "instance": "10.142.0.11:9100" 12:34:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:34:15 DEBUG "0.00043333333333066547" 12:34:15 DEBUG ] 12:34:15 DEBUG }, 12:34:15 DEBUG }, 12:34:15 DEBUG ] 12:34:15 DEBUG } 12:34:15 DEBUG "value": [ 12:34:15 DEBUG "value": [ 12:34:15 DEBUG } 12:34:15 DEBUG ] 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG ] 12:34:15 DEBUG } 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG } 12:34:15 DEBUG [...] 12:34:15 DEBUG [...] 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG "instance": "10.142.0.51:9100" 12:34:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:34:15 DEBUG }, 12:34:15 DEBUG }, 12:34:15 DEBUG "value": [ 12:34:15 DEBUG "value": [ 12:34:15 DEBUG 1735907655, 12:34:15 DEBUG 1735907655, 12:34:15 DEBUG "0.0006666666666670077" 12:34:15 DEBUG "0.1057909011923647" 12:34:15 DEBUG ] 12:34:15 DEBUG ] 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 DEBUG ] 12:34:15 DEBUG ] 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907655" 12:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907655" 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG ----- output ----- 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { 12:34:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:34:15 DEBUG "status": "success", 12:34:15 DEBUG ----- output ----- 12:34:15 DEBUG "data": { 12:34:15 DEBUG { 12:34:15 DEBUG "resultType": "vector", 12:34:15 DEBUG "status": "success", 12:34:15 DEBUG "result": [ 12:34:15 DEBUG "data": { 12:34:15 DEBUG { 12:34:15 DEBUG "resultType": "vector", 12:34:15 DEBUG "metric": { 12:34:15 DEBUG "result": [ 12:34:15 DEBUG "instance": "10.142.0.11:9100" 12:34:15 DEBUG { 12:34:15 DEBUG }, 12:34:15 DEBUG "metric": { 12:34:15 DEBUG "value": [ 12:34:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:34:15 DEBUG 12:34:15 DEBUG }, 12:34:15 DEBUG 12:34:15 DEBUG "value": [ 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG [...] 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG [...] 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG "instance": "10.142.0.51:9100" 12:34:15 DEBUG 12:34:15 DEBUG }, 12:34:15 DEBUG 12:34:15 DEBUG "value": [ 12:34:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:34:15 DEBUG 1735907655, 12:34:15 DEBUG }, 12:34:15 DEBUG "958337.0333333333" 12:34:15 DEBUG "value": [ 12:34:15 DEBUG ] 12:34:15 DEBUG 1735907655, 12:34:15 DEBUG } 12:34:15 DEBUG "14991818752" 12:34:15 DEBUG ] 12:34:15 DEBUG ] 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 DEBUG ] 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 INFO [http_cmd]: http status code OK 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG --- status code --- 12:34:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:34:15 DEBUG ----- output ----- 12:34:15 DEBUG --- http response --- 12:34:15 DEBUG { 12:34:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:34:15 DEBUG "status": "success", 12:34:15 DEBUG ----- output ----- 12:34:15 DEBUG "data": { 12:34:15 DEBUG { 12:34:15 DEBUG "resultType": "vector", 12:34:15 DEBUG "status": "success", 12:34:15 DEBUG "result": [ 12:34:15 DEBUG "data": { 12:34:15 DEBUG { 12:34:15 DEBUG "resultType": "vector", 12:34:15 DEBUG "metric": { 12:34:15 DEBUG "result": [ 12:34:15 DEBUG "pod": "ds-idrepo-0" 12:34:15 DEBUG { 12:34:15 DEBUG }, 12:34:15 DEBUG "metric": { 12:34:15 DEBUG "value": [ 12:34:15 DEBUG "instance": "10.142.0.11:9100" 12:34:15 DEBUG 12:34:15 DEBUG }, 12:34:15 DEBUG 12:34:15 DEBUG "value": [ 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG [...] 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG [...] 12:34:15 DEBUG 12:34:15 DEBUG 12:34:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:34:15 DEBUG 12:34:15 DEBUG }, 12:34:15 DEBUG 12:34:15 DEBUG "value": [ 12:34:15 DEBUG "instance": "10.142.0.51:9100" 12:34:15 DEBUG 1735907655, 12:34:15 DEBUG }, 12:34:15 DEBUG "758558720" 12:34:15 DEBUG "value": [ 12:34:15 DEBUG ] 12:34:15 DEBUG 1735907655, 12:34:15 DEBUG } 12:34:15 DEBUG "22937.6" 12:34:15 DEBUG ] 12:34:15 DEBUG ] 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:15 DEBUG ] 12:34:15 DEBUG } 12:34:15 DEBUG } 12:34:21 INFO 12:34:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:34:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:34:21 INFO [loop_until]: OK (rc = 0) 12:34:21 DEBUG --- stdout --- 12:34:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 158m 0% 3859Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 133m 0% 19049Mi 66% 12:34:21 DEBUG --- stderr --- 12:34:21 DEBUG 12:34:25 INFO 12:34:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:34:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:34:25 INFO [loop_until]: OK (rc = 0) 12:34:25 DEBUG --- stdout --- 12:34:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 81m 13478Mi rcs-645c86fdd6-j4rsk 47m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:34:25 DEBUG --- stderr --- 12:34:25 DEBUG 12:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907714" 12:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907714" 12:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907714" 12:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907714" 12:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907714" 12:35:14 INFO [http_cmd]: http status code OK 12:35:14 DEBUG --- status code --- 12:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:14 DEBUG --- http response --- 12:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:14 INFO [http_cmd]: http status code OK 12:35:14 DEBUG --- status code --- 12:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:14 DEBUG --- http response --- 12:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:14 INFO [http_cmd]: http status code OK 12:35:14 DEBUG --- status code --- 12:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:14 DEBUG --- http response --- 12:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:14 ERROR [http_cmd]: ERROR 12:35:14 ERROR --- status code --- 12:35:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:35:14 ERROR --- http response --- 12:35:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:35:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:35:14 INFO [http_cmd]: http status code OK 12:35:14 DEBUG --- status code --- 12:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:14 DEBUG --- http response --- 12:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907714" 12:35:14 INFO [http_cmd]: http status code OK 12:35:14 DEBUG --- status code --- 12:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:14 DEBUG --- http response --- 12:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907714" 12:35:14 INFO [http_cmd]: http status code OK 12:35:14 DEBUG --- status code --- 12:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:14 DEBUG --- http response --- 12:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG { 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG "data": { 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "result": [ 12:35:15 DEBUG { 12:35:15 DEBUG "metric": { 12:35:15 DEBUG "pod": "ds-idrepo-0" 12:35:15 DEBUG }, 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG [...] 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:35:15 DEBUG }, 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG "0" 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG "data": { 12:35:15 DEBUG { 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG "result": [ 12:35:15 DEBUG "data": { 12:35:15 DEBUG { 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "metric": { 12:35:15 DEBUG "result": [ 12:35:15 DEBUG "pod": "ds-idrepo-0" 12:35:15 DEBUG { 12:35:15 DEBUG }, 12:35:15 DEBUG "metric": { 12:35:15 DEBUG "value": [ 12:35:15 DEBUG "pod": "ds-idrepo-0" 12:35:15 DEBUG 12:35:15 DEBUG }, 12:35:15 DEBUG 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG [...] 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG [...] 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG 12:35:15 DEBUG }, 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG 12:35:15 DEBUG "value": [ 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG "pod": "ds-cts-0" 12:35:15 DEBUG 1735907715, 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- http response --- 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG }, 12:35:15 DEBUG "910.8161252404634" 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:35:15 DEBUG "value": [ 12:35:15 DEBUG ] 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG } 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:35:15 DEBUG "0" 12:35:15 DEBUG ] 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG { 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG "data": { 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG { 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 DEBUG { 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "data": { 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG ] 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG "result": [ 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "data": { 12:35:15 DEBUG } 12:35:15 DEBUG "data": { 12:35:15 DEBUG { 12:35:15 DEBUG "result": [ 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG } 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "metric": { 12:35:15 DEBUG { 12:35:15 DEBUG "result": [ 12:35:15 DEBUG "result": [ 12:35:15 DEBUG "instance": "10.142.0.11:9100" 12:35:15 DEBUG "metric": { 12:35:15 DEBUG { 12:35:15 DEBUG { 12:35:15 DEBUG }, 12:35:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:35:15 DEBUG "metric": { 12:35:15 DEBUG "metric": { 12:35:15 DEBUG "value": [ 12:35:15 DEBUG }, 12:35:15 DEBUG "instance": "10.142.0.11:9100" 12:35:15 DEBUG }, 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 12:35:15 DEBUG "instance": "10.142.0.11:9100" 12:35:15 DEBUG 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG }, 12:35:15 DEBUG [...] 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG "instance": "10.142.0.51:9100" 12:35:15 DEBUG [...] 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG }, 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG "0.000466666666664158" 12:35:15 DEBUG 12:35:15 DEBUG [...] 12:35:15 DEBUG 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG } 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG [...] 12:35:15 DEBUG "instance": "10.142.0.51:9100" 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG }, 12:35:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:35:15 DEBUG 12:35:15 DEBUG "value": [ 12:35:15 DEBUG }, 12:35:15 DEBUG 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG "value": [ 12:35:15 DEBUG "instance": "10.142.0.51:9100" 12:35:15 DEBUG "89528.30000000002" 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG }, 12:35:15 DEBUG ] 12:35:15 DEBUG "0.10156804228410418" 12:35:15 DEBUG "value": [ 12:35:15 DEBUG } 12:35:15 DEBUG ] 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG "0.00033333333333397756" 12:35:15 DEBUG } 12:35:15 DEBUG ] 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907715" 12:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907715" 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG { 12:35:15 DEBUG { 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG "data": { 12:35:15 DEBUG "data": { 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "result": [ 12:35:15 DEBUG "result": [ 12:35:15 DEBUG { 12:35:15 DEBUG { 12:35:15 DEBUG "metric": { 12:35:15 DEBUG "metric": { 12:35:15 DEBUG "instance": "10.142.0.11:9100" 12:35:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:35:15 DEBUG }, 12:35:15 DEBUG }, 12:35:15 DEBUG "value": [ 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG [...] 12:35:15 DEBUG [...] 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG "instance": "10.142.0.51:9100" 12:35:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:35:15 DEBUG }, 12:35:15 DEBUG }, 12:35:15 DEBUG "value": [ 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG "72607.9" 12:35:15 DEBUG "14985150464" 12:35:15 DEBUG ] 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 DEBUG ] 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 INFO [http_cmd]: http status code OK 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG --- status code --- 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG --- http response --- 12:35:15 DEBUG { 12:35:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG ----- output ----- 12:35:15 DEBUG "data": { 12:35:15 DEBUG { 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "status": "success", 12:35:15 DEBUG "result": [ 12:35:15 DEBUG "data": { 12:35:15 DEBUG { 12:35:15 DEBUG "resultType": "vector", 12:35:15 DEBUG "metric": { 12:35:15 DEBUG "result": [ 12:35:15 DEBUG "pod": "ds-idrepo-0" 12:35:15 DEBUG }, 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 12:35:15 DEBUG { 12:35:15 DEBUG 12:35:15 DEBUG "metric": { 12:35:15 DEBUG 12:35:15 DEBUG "instance": "10.142.0.11:9100" 12:35:15 DEBUG [...] 12:35:15 DEBUG }, 12:35:15 DEBUG 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG 12:35:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:35:15 DEBUG 12:35:15 DEBUG }, 12:35:15 DEBUG [...] 12:35:15 DEBUG "value": [ 12:35:15 DEBUG 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG 12:35:15 DEBUG "758558720" 12:35:15 DEBUG 12:35:15 DEBUG ] 12:35:15 DEBUG "instance": "10.142.0.51:9100" 12:35:15 DEBUG } 12:35:15 DEBUG }, 12:35:15 DEBUG ] 12:35:15 DEBUG "value": [ 12:35:15 DEBUG } 12:35:15 DEBUG 1735907715, 12:35:15 DEBUG } 12:35:15 DEBUG "19114.666666666664" 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG ] 12:35:15 DEBUG } 12:35:15 DEBUG } 12:35:21 INFO 12:35:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:35:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:35:21 INFO [loop_until]: OK (rc = 0) 12:35:21 DEBUG --- stdout --- 12:35:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3869Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 89m 0% 19048Mi 66% 12:35:21 DEBUG --- stderr --- 12:35:21 DEBUG 12:35:25 INFO 12:35:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:35:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:35:25 INFO [loop_until]: OK (rc = 0) 12:35:25 DEBUG --- stdout --- 12:35:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 23m 13479Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:35:25 DEBUG --- stderr --- 12:35:25 DEBUG 12:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907774" 12:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907774" 12:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907774" 12:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907774" 12:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907774" 12:36:14 INFO [http_cmd]: http status code OK 12:36:14 DEBUG --- status code --- 12:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:14 DEBUG --- http response --- 12:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:14 INFO [http_cmd]: http status code OK 12:36:14 INFO [http_cmd]: http status code OK 12:36:14 ERROR [http_cmd]: ERROR 12:36:14 INFO [http_cmd]: http status code OK 12:36:14 DEBUG --- status code --- 12:36:14 DEBUG --- status code --- 12:36:14 ERROR --- status code --- 12:36:14 DEBUG --- status code --- 12:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:14 DEBUG --- http response --- 12:36:14 DEBUG --- http response --- 12:36:14 ERROR --- http response --- 12:36:14 DEBUG --- http response --- 12:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907774" 12:36:14 INFO [http_cmd]: http status code OK 12:36:14 DEBUG --- status code --- 12:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:14 DEBUG --- http response --- 12:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907774" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG { 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG "data": { 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG "result": [ 12:36:15 DEBUG { 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "pod": "ds-idrepo-0" 12:36:15 DEBUG }, 12:36:15 DEBUG "value": [ 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG [...] 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:36:15 DEBUG }, 12:36:15 DEBUG "value": [ 12:36:15 DEBUG 1735907775, 12:36:15 DEBUG "0" 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907775" 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907775" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:36:15 DEBUG "data": { 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG { 12:36:15 DEBUG "result": [ 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG { 12:36:15 DEBUG "data": { 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG "pod": "ds-idrepo-0" 12:36:15 DEBUG "result": [ 12:36:15 DEBUG }, 12:36:15 DEBUG { 12:36:15 DEBUG "value": [ 12:36:15 DEBUG "metric": { 12:36:15 DEBUG 12:36:15 DEBUG "pod": "ds-idrepo-0" 12:36:15 DEBUG 12:36:15 DEBUG }, 12:36:15 DEBUG 12:36:15 DEBUG "value": [ 12:36:15 DEBUG [...] 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG [...] 12:36:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG 12:36:15 DEBUG }, 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG 12:36:15 DEBUG "value": [ 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG 12:36:15 DEBUG 1735907775, 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:36:15 DEBUG "0.0005964122635500291" 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG }, 12:36:15 DEBUG ] 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG "value": [ 12:36:15 DEBUG } 12:36:15 DEBUG { 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG 1735907775, 12:36:15 DEBUG ] 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG "133.59425962165687" 12:36:15 DEBUG } 12:36:15 DEBUG "data": { 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG } 12:36:15 DEBUG "result": [ 12:36:15 DEBUG { 12:36:15 DEBUG ] 12:36:15 DEBUG { 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG } 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "data": { 12:36:15 DEBUG } 12:36:15 DEBUG "instance": "10.142.0.11:9100" 12:36:15 DEBUG "resultType": "vector", 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG }, 12:36:15 DEBUG "result": [ 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG "value": [ 12:36:15 DEBUG { 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "instance": "10.142.0.11:9100" 12:36:15 DEBUG 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG }, 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG "value": [ 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:36:15 DEBUG [...] 12:36:15 DEBUG [...] 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG { 12:36:15 DEBUG { 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG "data": { 12:36:15 DEBUG "data": { 12:36:15 DEBUG "instance": "10.142.0.51:9100" 12:36:15 DEBUG "instance": "10.142.0.51:9100" 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG }, 12:36:15 DEBUG }, 12:36:15 DEBUG "result": [ 12:36:15 DEBUG "result": [ 12:36:15 DEBUG "value": [ 12:36:15 DEBUG "value": [ 12:36:15 DEBUG { 12:36:15 DEBUG { 12:36:15 DEBUG 1735907775, 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907775" 12:36:15 DEBUG 1735907775, 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907775" 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "0.0004333333333382446" 12:36:15 DEBUG "93110.59999999999" 12:36:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:36:15 DEBUG "instance": "10.142.0.11:9100" 12:36:15 DEBUG ] 12:36:15 DEBUG ] 12:36:15 DEBUG }, 12:36:15 DEBUG }, 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 DEBUG "value": [ 12:36:15 DEBUG "value": [ 12:36:15 DEBUG ] 12:36:15 DEBUG ] 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG [...] 12:36:15 DEBUG [...] 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:36:15 DEBUG "instance": "10.142.0.51:9100" 12:36:15 DEBUG }, 12:36:15 DEBUG }, 12:36:15 DEBUG "value": [ 12:36:15 DEBUG "value": [ 12:36:15 DEBUG 1735907775, 12:36:15 DEBUG 1735907775, 12:36:15 DEBUG "0.0921566653445742" 12:36:15 DEBUG "0.00033333333333350387" 12:36:15 DEBUG ] 12:36:15 DEBUG ] 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907775" 12:36:15 DEBUG } 12:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907775" 12:36:15 DEBUG } 12:36:15 DEBUG ] 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG { 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG "data": { 12:36:15 DEBUG { 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG "result": [ 12:36:15 DEBUG "data": { 12:36:15 DEBUG { 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "result": [ 12:36:15 DEBUG "instance": "10.142.0.11:9100" 12:36:15 DEBUG { 12:36:15 DEBUG }, 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "value": [ 12:36:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:36:15 DEBUG 12:36:15 DEBUG }, 12:36:15 DEBUG 12:36:15 DEBUG "value": [ 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG [...] 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG [...] 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG "instance": "10.142.0.51:9100" 12:36:15 DEBUG 12:36:15 DEBUG }, 12:36:15 DEBUG 12:36:15 DEBUG "value": [ 12:36:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:36:15 DEBUG 1735907775, 12:36:15 DEBUG }, 12:36:15 DEBUG "76310.73333333332" 12:36:15 DEBUG "value": [ 12:36:15 DEBUG ] 12:36:15 DEBUG 1735907775, 12:36:15 DEBUG } 12:36:15 DEBUG "14982459392" 12:36:15 DEBUG ] 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG { 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG "data": { 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG "result": [ 12:36:15 DEBUG { 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "pod": "ds-idrepo-0" 12:36:15 DEBUG }, 12:36:15 DEBUG "value": [ 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG [...] 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:36:15 DEBUG }, 12:36:15 DEBUG "value": [ 12:36:15 DEBUG 1735907775, 12:36:15 DEBUG "758558720" 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:15 INFO [http_cmd]: http status code OK 12:36:15 DEBUG --- status code --- 12:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:36:15 DEBUG --- http response --- 12:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:36:15 DEBUG ----- output ----- 12:36:15 DEBUG { 12:36:15 DEBUG "status": "success", 12:36:15 DEBUG "data": { 12:36:15 DEBUG "resultType": "vector", 12:36:15 DEBUG "result": [ 12:36:15 DEBUG { 12:36:15 DEBUG "metric": { 12:36:15 DEBUG "instance": "10.142.0.11:9100" 12:36:15 DEBUG }, 12:36:15 DEBUG "value": [ 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG [...] 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG 12:36:15 DEBUG "instance": "10.142.0.51:9100" 12:36:15 DEBUG }, 12:36:15 DEBUG "value": [ 12:36:15 DEBUG 1735907775, 12:36:15 DEBUG "25395.199999999997" 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG ] 12:36:15 DEBUG } 12:36:15 DEBUG } 12:36:21 INFO 12:36:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:36:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:36:21 INFO [loop_until]: OK (rc = 0) 12:36:21 DEBUG --- stdout --- 12:36:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3860Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 87m 0% 19050Mi 66% 12:36:21 DEBUG --- stderr --- 12:36:21 DEBUG 12:36:25 INFO 12:36:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:36:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:36:25 INFO [loop_until]: OK (rc = 0) 12:36:25 DEBUG --- stdout --- 12:36:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13478Mi rcs-645c86fdd6-j4rsk 1m 821Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:36:25 DEBUG --- stderr --- 12:36:25 DEBUG 12:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907834" 12:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907834" 12:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907834" 12:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907834" 12:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907834" 12:37:14 ERROR [http_cmd]: ERROR 12:37:14 ERROR --- status code --- 12:37:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:37:14 ERROR --- http response --- 12:37:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:37:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:37:14 INFO [http_cmd]: http status code OK 12:37:14 DEBUG --- status code --- 12:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:14 DEBUG --- http response --- 12:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:14 INFO [http_cmd]: http status code OK 12:37:14 DEBUG --- status code --- 12:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:14 DEBUG --- http response --- 12:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:14 INFO [http_cmd]: http status code OK 12:37:14 DEBUG --- status code --- 12:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:14 DEBUG --- http response --- 12:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:14 INFO [http_cmd]: http status code OK 12:37:14 DEBUG --- status code --- 12:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:14 DEBUG --- http response --- 12:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907834" 12:37:14 INFO [http_cmd]: http status code OK 12:37:14 DEBUG --- status code --- 12:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:14 DEBUG --- http response --- 12:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735907835, "0" ] } ] } } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735907835, "0" ] } ] } } 12:37:15 DEBUG ----- output ----- 12:37:15 DEBUG { 12:37:15 DEBUG "status": "success", 12:37:15 DEBUG "data": { 12:37:15 DEBUG "resultType": "vector", 12:37:15 DEBUG "result": [ 12:37:15 DEBUG { 12:37:15 DEBUG "metric": { 12:37:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:37:15 DEBUG }, 12:37:15 DEBUG "value": [ 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG [...] 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG "pod": "ds-cts-0" 12:37:15 DEBUG }, 12:37:15 DEBUG "value": [ 12:37:15 DEBUG 1735907835, 12:37:15 DEBUG "0.0016142056506808068" 12:37:15 DEBUG ] 12:37:15 DEBUG } 12:37:15 DEBUG ] 12:37:15 DEBUG } 12:37:15 DEBUG } 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907835" 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907835" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907835" 12:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:37:15 DEBUG ----- output ----- 12:37:15 DEBUG { 12:37:15 DEBUG "status": "success", 12:37:15 DEBUG "data": { 12:37:15 DEBUG "resultType": "vector", 12:37:15 DEBUG "result": [ 12:37:15 DEBUG { 12:37:15 DEBUG "metric": { 12:37:15 DEBUG "instance": "10.142.0.11:9100" 12:37:15 DEBUG }, 12:37:15 DEBUG "value": [ 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG [...] 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG 12:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:37:15 DEBUG 12:37:15 DEBUG ----- output ----- 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907835" 12:37:15 DEBUG "instance": "10.142.0.51:9100" 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG { 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG }, 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG "status": "success", 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG "value": [ 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG "data": { 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG 1735907835, 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG "resultType": "vector", 12:37:15 DEBUG "0.0011666666666656055" 12:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:37:15 DEBUG "result": [ 12:37:15 DEBUG ] 12:37:15 DEBUG ----- output ----- 12:37:15 DEBUG ----- output ----- 12:37:15 DEBUG { 12:37:15 DEBUG } 12:37:15 DEBUG { 12:37:15 DEBUG { 12:37:15 DEBUG "metric": { 12:37:15 DEBUG ] 12:37:15 DEBUG "status": "success", 12:37:15 DEBUG "status": "success", 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG "instance": "10.142.0.11:9100" 12:37:15 DEBUG } 12:37:15 DEBUG "data": { 12:37:15 DEBUG "data": { 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG }, 12:37:15 DEBUG } 12:37:15 DEBUG "resultType": "vector", 12:37:15 DEBUG "resultType": "vector", 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG "value": [ 12:37:15 DEBUG "result": [ 12:37:15 DEBUG "result": [ 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG 12:37:15 DEBUG { 12:37:15 DEBUG { 12:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:37:15 DEBUG 12:37:15 DEBUG "metric": { 12:37:15 DEBUG "metric": { 12:37:15 DEBUG ----- output ----- 12:37:15 DEBUG ----- output ----- 12:37:15 DEBUG 12:37:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:37:15 DEBUG "instance": "10.142.0.11:9100" 12:37:15 DEBUG { 12:37:15 DEBUG { 12:37:15 DEBUG [...] 12:37:15 DEBUG }, 12:37:15 DEBUG }, 12:37:15 DEBUG "status": "success", 12:37:15 DEBUG "status": "success", 12:37:15 DEBUG 12:37:15 DEBUG "value": [ 12:37:15 DEBUG "value": [ 12:37:15 DEBUG "data": { 12:37:15 DEBUG "data": { 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG "resultType": "vector", 12:37:15 DEBUG "resultType": "vector", 12:37:15 DEBUG 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG "result": [ 12:37:15 DEBUG "result": [ 12:37:15 DEBUG "instance": "10.142.0.51:9100" 12:37:15 DEBUG }, 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG { 12:37:15 DEBUG { 12:37:15 INFO [http_cmd]: http status code OK 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG "value": [ 12:37:15 DEBUG [...] 12:37:15 DEBUG [...] 12:37:15 DEBUG "metric": { 12:37:15 DEBUG "metric": { 12:37:15 DEBUG --- status code --- 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG 1735907835, 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG "415243.8333333332" 12:37:15 DEBUG 12:37:15 DEBUG "instance": "10.142.0.11:9100" 12:37:15 DEBUG 12:37:15 DEBUG }, 12:37:15 DEBUG --- http response --- 12:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:37:15 DEBUG ] 12:37:15 DEBUG } 12:37:15 DEBUG ] 12:37:15 DEBUG } 12:37:15 DEBUG "value": [ 12:37:15 DEBUG ----- output ----- 12:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:37:15 DEBUG 12:37:15 DEBUG }, 12:37:15 DEBUG "instance": "10.142.0.51:9100" 12:37:15 DEBUG } 12:37:15 DEBUG 12:37:15 DEBUG { 12:37:15 DEBUG ----- output ----- 12:37:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:37:15 DEBUG "value": [ 12:37:15 DEBUG }, 12:37:15 DEBUG 12:37:15 DEBUG "status": "success", 12:37:15 DEBUG { 12:37:15 DEBUG }, 12:37:15 DEBUG 12:37:15 DEBUG "value": [ 12:37:15 DEBUG 12:37:15 DEBUG "data": { 12:37:15 DEBUG "status": "success", 12:37:15 DEBUG "value": [ 12:37:15 DEBUG 12:37:15 DEBUG 1735907835, 12:37:15 DEBUG [...] 12:37:15 DEBUG "resultType": "vector", 12:37:15 DEBUG "data": { 12:37:15 DEBUG 1735907835, 12:37:15 DEBUG "0.058793698619269294" 12:37:15 DEBUG "0.001000000000000038" 12:37:15 DEBUG 12:37:15 DEBUG "result": [ 12:37:15 DEBUG "resultType": "vector", 12:37:15 DEBUG 12:37:15 DEBUG ] 12:37:15 DEBUG ] 12:37:15 DEBUG 12:37:15 DEBUG { 12:37:15 DEBUG "result": [ 12:37:15 DEBUG [...] 12:37:15 DEBUG 12:37:15 DEBUG } 12:37:15 DEBUG 12:37:15 DEBUG "metric": { 12:37:15 DEBUG { 12:37:15 DEBUG } 12:37:15 DEBUG 12:37:15 DEBUG ] 12:37:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:37:15 DEBUG "pod": "ds-idrepo-0" 12:37:15 DEBUG "metric": { 12:37:15 DEBUG ] 12:37:15 DEBUG 12:37:15 DEBUG } 12:37:15 DEBUG }, 12:37:15 DEBUG }, 12:37:15 DEBUG "instance": "10.142.0.11:9100" 12:37:15 DEBUG } 12:37:15 DEBUG "instance": "10.142.0.51:9100" 12:37:15 DEBUG } 12:37:15 DEBUG "value": [ 12:37:15 DEBUG "value": [ 12:37:15 DEBUG }, 12:37:15 DEBUG } 12:37:15 DEBUG }, 12:37:15 DEBUG 1735907835, 12:37:15 DEBUG 12:37:15 DEBUG "value": [ 12:37:15 DEBUG "value": [ 12:37:15 DEBUG "14986551296" 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG 1735907835, 12:37:15 DEBUG ] 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG "398330.0666666666" 12:37:15 DEBUG } 12:37:15 DEBUG [...] 12:37:15 DEBUG 12:37:15 DEBUG ] 12:37:15 DEBUG ] 12:37:15 DEBUG 12:37:15 DEBUG [...] 12:37:15 DEBUG } 12:37:15 DEBUG } 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG ] 12:37:15 DEBUG } 12:37:15 DEBUG 12:37:15 DEBUG 12:37:15 DEBUG } 12:37:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:37:15 DEBUG 12:37:15 DEBUG } 12:37:15 DEBUG }, 12:37:15 DEBUG "instance": "10.142.0.51:9100" 12:37:15 DEBUG "value": [ 12:37:15 DEBUG }, 12:37:15 DEBUG 1735907835, 12:37:15 DEBUG "value": [ 12:37:15 DEBUG "758558720" 12:37:15 DEBUG 1735907835, 12:37:15 DEBUG ] 12:37:15 DEBUG "31402.666666666664" 12:37:15 DEBUG } 12:37:15 DEBUG ] 12:37:15 DEBUG ] 12:37:15 DEBUG } 12:37:15 DEBUG } 12:37:15 DEBUG ] 12:37:15 DEBUG } 12:37:15 DEBUG } 12:37:15 DEBUG } 12:37:21 INFO 12:37:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:37:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:37:21 INFO [loop_until]: OK (rc = 0) 12:37:21 DEBUG --- stdout --- 12:37:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 131m 0% 3860Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 96m 0% 19052Mi 66% 12:37:21 DEBUG --- stderr --- 12:37:21 DEBUG 12:37:25 INFO 12:37:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:37:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:37:25 INFO [loop_until]: OK (rc = 0) 12:37:25 DEBUG --- stdout --- 12:37:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13479Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:37:25 DEBUG --- stderr --- 12:37:25 DEBUG 12:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907894" 12:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907894" 12:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907894" 12:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907894" 12:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907894" 12:38:14 INFO [http_cmd]: http status code OK 12:38:14 DEBUG --- status code --- 12:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:14 DEBUG --- http response --- 12:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:14 ERROR [http_cmd]: ERROR 12:38:14 INFO [http_cmd]: http status code OK 12:38:14 ERROR --- status code --- 12:38:14 DEBUG --- status code --- 12:38:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:14 ERROR --- http response --- 12:38:14 DEBUG --- http response --- 12:38:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:38:14 INFO [http_cmd]: http status code OK 12:38:14 INFO [http_cmd]: http status code OK 12:38:14 DEBUG --- status code --- 12:38:14 DEBUG --- status code --- 12:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:14 DEBUG --- http response --- 12:38:14 DEBUG --- http response --- 12:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907894" 12:38:14 INFO [http_cmd]: http status code OK 12:38:14 DEBUG --- status code --- 12:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:14 DEBUG --- http response --- 12:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG --- http response --- 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG --- http response --- 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- http response --- 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG { 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG "data": { 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "result": [ 12:38:15 DEBUG { 12:38:15 DEBUG "metric": { 12:38:15 DEBUG "pod": "ds-idrepo-0" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG [...] 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG "0" 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG { 12:38:15 DEBUG { 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG "data": { 12:38:15 DEBUG "data": { 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "result": [ 12:38:15 DEBUG "result": [ 12:38:15 DEBUG { 12:38:15 DEBUG { 12:38:15 DEBUG "metric": { 12:38:15 DEBUG "metric": { 12:38:15 DEBUG "pod": "ds-idrepo-0" 12:38:15 DEBUG "pod": "ds-idrepo-0" 12:38:15 DEBUG }, 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG [...] 12:38:15 DEBUG [...] 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:38:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:38:15 DEBUG }, 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG "0.0006489833271338738" 12:38:15 DEBUG "351.4170503617697" 12:38:15 DEBUG ] 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 DEBUG ] 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG { 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG "data": { 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "result": [ 12:38:15 DEBUG { 12:38:15 DEBUG "metric": { 12:38:15 DEBUG "instance": "10.142.0.11:9100" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG [...] 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG "instance": "10.142.0.51:9100" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG "0.0005333333333292482" 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907895" 12:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907895" 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG { 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG "data": { 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "result": [ 12:38:15 DEBUG { 12:38:15 DEBUG "metric": { 12:38:15 DEBUG "instance": "10.142.0.11:9100" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG [...] 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG "instance": "10.142.0.51:9100" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG "1252383.3666666667" 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG ] 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG } 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG } 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG { 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG "data": { 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "result": [ 12:38:15 DEBUG { 12:38:15 DEBUG "metric": { 12:38:15 DEBUG "instance": "10.142.0.11:9100" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG [...] 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG "instance": "10.142.0.51:9100" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG "0.000666666666666534" 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG { 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG "data": { 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "result": [ 12:38:15 DEBUG { 12:38:15 DEBUG "metric": { 12:38:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG 12:38:15 DEBUG --- status code --- 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG [...] 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:38:15 DEBUG 12:38:15 DEBUG { 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG { 12:38:15 DEBUG }, 12:38:15 DEBUG "data": { 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG "value": [ 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "data": { 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG "result": [ 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "0.06709193310998733" 12:38:15 DEBUG { 12:38:15 DEBUG "result": [ 12:38:15 DEBUG ] 12:38:15 DEBUG "metric": { 12:38:15 DEBUG { 12:38:15 DEBUG } 12:38:15 DEBUG "instance": "10.142.0.11:9100" 12:38:15 DEBUG "metric": { 12:38:15 DEBUG ] 12:38:15 DEBUG }, 12:38:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:38:15 DEBUG } 12:38:15 DEBUG "value": [ 12:38:15 DEBUG }, 12:38:15 DEBUG } 12:38:15 DEBUG 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG [...] 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG [...] 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG "instance": "10.142.0.51:9100" 12:38:15 DEBUG 12:38:15 DEBUG }, 12:38:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:38:15 DEBUG "value": [ 12:38:15 DEBUG }, 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG "12288" 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG ] 12:38:15 DEBUG "14990024704" 12:38:15 DEBUG } 12:38:15 DEBUG ] 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG { 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG "data": { 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG "result": [ 12:38:15 DEBUG { 12:38:15 INFO [http_cmd]: http status code OK 12:38:15 DEBUG "metric": { 12:38:15 DEBUG --- status code --- 12:38:15 DEBUG "instance": "10.142.0.11:9100" 12:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:38:15 DEBUG }, 12:38:15 DEBUG --- http response --- 12:38:15 DEBUG "value": [ 12:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:38:15 DEBUG 12:38:15 DEBUG ----- output ----- 12:38:15 DEBUG 12:38:15 DEBUG { 12:38:15 DEBUG 12:38:15 DEBUG "status": "success", 12:38:15 DEBUG [...] 12:38:15 DEBUG "data": { 12:38:15 DEBUG 12:38:15 DEBUG "resultType": "vector", 12:38:15 DEBUG 12:38:15 DEBUG "result": [ 12:38:15 DEBUG 12:38:15 DEBUG { 12:38:15 DEBUG "instance": "10.142.0.51:9100" 12:38:15 DEBUG "metric": { 12:38:15 DEBUG }, 12:38:15 DEBUG "pod": "ds-idrepo-0" 12:38:15 DEBUG "value": [ 12:38:15 DEBUG }, 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG "1234892.7333333332" 12:38:15 DEBUG 12:38:15 DEBUG ] 12:38:15 DEBUG 12:38:15 DEBUG } 12:38:15 DEBUG 12:38:15 DEBUG ] 12:38:15 DEBUG [...] 12:38:15 DEBUG } 12:38:15 DEBUG 12:38:15 DEBUG } 12:38:15 DEBUG 12:38:15 DEBUG 12:38:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:38:15 DEBUG }, 12:38:15 DEBUG "value": [ 12:38:15 DEBUG 1735907895, 12:38:15 DEBUG "758558720" 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG ] 12:38:15 DEBUG } 12:38:15 DEBUG } 12:38:21 INFO 12:38:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:38:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:38:21 INFO [loop_until]: OK (rc = 0) 12:38:21 DEBUG --- stdout --- 12:38:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 174m 1% 3862Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 141m 0% 19052Mi 66% 12:38:21 DEBUG --- stderr --- 12:38:21 DEBUG 12:38:25 INFO 12:38:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:38:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:38:25 INFO [loop_until]: OK (rc = 0) 12:38:25 DEBUG --- stdout --- 12:38:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 53m 13479Mi rcs-645c86fdd6-j4rsk 66m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:38:25 DEBUG --- stderr --- 12:38:25 DEBUG 12:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907954" 12:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735907954" 12:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907954" 12:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907954" 12:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907954" 12:39:14 INFO [http_cmd]: http status code OK 12:39:14 DEBUG --- status code --- 12:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:14 DEBUG --- http response --- 12:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:14 INFO [http_cmd]: http status code OK 12:39:14 DEBUG --- status code --- 12:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:14 DEBUG --- http response --- 12:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:14 INFO [http_cmd]: http status code OK 12:39:14 DEBUG --- status code --- 12:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:14 DEBUG --- http response --- 12:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:14 ERROR [http_cmd]: ERROR 12:39:14 ERROR --- status code --- 12:39:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:39:14 ERROR --- http response --- 12:39:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:39:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:39:14 INFO [http_cmd]: http status code OK 12:39:14 DEBUG --- status code --- 12:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:14 DEBUG --- http response --- 12:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907954" 12:39:14 INFO [http_cmd]: http status code OK 12:39:14 DEBUG --- status code --- 12:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:14 DEBUG --- http response --- 12:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG { 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG "data": { 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG "result": [ 12:39:15 DEBUG { 12:39:15 DEBUG "metric": { 12:39:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG [...] 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG "0" 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG { 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG "data": { 12:39:15 DEBUG "resultType": "vector", 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG "result": [ 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG { 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG "metric": { 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG }, 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG "value": [ 12:39:15 DEBUG { 12:39:15 DEBUG 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG 12:39:15 DEBUG "data": { 12:39:15 DEBUG 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG [...] 12:39:15 DEBUG "result": [ 12:39:15 DEBUG 12:39:15 DEBUG { 12:39:15 DEBUG 12:39:15 DEBUG "metric": { 12:39:15 DEBUG 12:39:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:39:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:39:15 DEBUG }, 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG 12:39:15 DEBUG "124.22284899766477" 12:39:15 DEBUG 12:39:15 DEBUG ] 12:39:15 DEBUG [...] 12:39:15 DEBUG } 12:39:15 DEBUG 12:39:15 DEBUG ] 12:39:15 DEBUG 12:39:15 DEBUG } 12:39:15 DEBUG 12:39:15 DEBUG } 12:39:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG "0.0005858732902574025" 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG { 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG "data": { 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG "result": [ 12:39:15 DEBUG { 12:39:15 DEBUG "metric": { 12:39:15 DEBUG "instance": "10.142.0.11:9100" 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG [...] 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG "instance": "10.142.0.51:9100" 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG "0.0005000000000071243" 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG { 12:39:15 DEBUG { 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG "data": { 12:39:15 DEBUG "data": { 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG "result": [ 12:39:15 DEBUG "result": [ 12:39:15 DEBUG { 12:39:15 DEBUG { 12:39:15 DEBUG "metric": { 12:39:15 DEBUG "metric": { 12:39:15 DEBUG "instance": "10.142.0.11:9100" 12:39:15 DEBUG "instance": "10.142.0.11:9100" 12:39:15 DEBUG }, 12:39:15 DEBUG }, 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG "value": [ 12:39:15 DEBUG "value": [ 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG [...] 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG [...] 12:39:15 DEBUG { 12:39:15 DEBUG 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG "data": { 12:39:15 DEBUG 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG "instance": "10.142.0.51:9100" 12:39:15 DEBUG { 12:39:15 DEBUG "instance": "10.142.0.51:9100" 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG "result": [ 12:39:15 DEBUG { 12:39:15 DEBUG "metric": { 12:39:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG }, 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG }, 12:39:15 DEBUG }, 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG "value": [ 12:39:15 DEBUG "data": { 12:39:15 DEBUG "value": [ 12:39:15 DEBUG "value": [ 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG 12:39:15 DEBUG { 12:39:15 DEBUG "0" 12:39:15 DEBUG "result": [ 12:39:15 DEBUG "1246402.066666667" 12:39:15 DEBUG 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG ] 12:39:15 DEBUG { 12:39:15 DEBUG ] 12:39:15 DEBUG 12:39:15 DEBUG "data": { 12:39:15 DEBUG } 12:39:15 DEBUG "metric": { 12:39:15 DEBUG } 12:39:15 DEBUG [...] 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG ] 12:39:15 DEBUG "instance": "10.142.0.11:9100" 12:39:15 DEBUG ] 12:39:15 DEBUG 12:39:15 DEBUG "result": [ 12:39:15 DEBUG } 12:39:15 DEBUG }, 12:39:15 DEBUG } 12:39:15 DEBUG 12:39:15 DEBUG { 12:39:15 DEBUG } 12:39:15 DEBUG "value": [ 12:39:15 DEBUG } 12:39:15 DEBUG 12:39:15 DEBUG "metric": { 12:39:15 DEBUG 12:39:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:39:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:39:15 DEBUG 12:39:15 DEBUG }, 12:39:15 DEBUG }, 12:39:15 DEBUG 12:39:15 DEBUG "value": [ 12:39:15 DEBUG "value": [ 12:39:15 DEBUG [...] 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG "0.033472029261803156" 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG ] 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG } 12:39:15 DEBUG [...] 12:39:15 DEBUG "instance": "10.142.0.51:9100" 12:39:15 DEBUG ] 12:39:15 DEBUG 12:39:15 DEBUG }, 12:39:15 DEBUG } 12:39:15 DEBUG 12:39:15 DEBUG "value": [ 12:39:15 DEBUG } 12:39:15 DEBUG 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:39:15 DEBUG "19251.2" 12:39:15 DEBUG }, 12:39:15 DEBUG ] 12:39:15 DEBUG "value": [ 12:39:15 DEBUG } 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG ] 12:39:15 DEBUG "14988066816" 12:39:15 DEBUG } 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG } 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG } 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735907955" 12:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735907955" 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG { 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG "data": { 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG "result": [ 12:39:15 DEBUG { 12:39:15 DEBUG "metric": { 12:39:15 DEBUG "instance": "10.142.0.11:9100" 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG [...] 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG "instance": "10.142.0.51:9100" 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG "1229000.6999999997" 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG } 12:39:15 INFO [http_cmd]: http status code OK 12:39:15 DEBUG --- status code --- 12:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:39:15 DEBUG --- http response --- 12:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:39:15 DEBUG ----- output ----- 12:39:15 DEBUG { 12:39:15 DEBUG "status": "success", 12:39:15 DEBUG "data": { 12:39:15 DEBUG "resultType": "vector", 12:39:15 DEBUG "result": [ 12:39:15 DEBUG { 12:39:15 DEBUG "metric": { 12:39:15 DEBUG "pod": "ds-idrepo-0" 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG [...] 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG 12:39:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:39:15 DEBUG }, 12:39:15 DEBUG "value": [ 12:39:15 DEBUG 1735907955, 12:39:15 DEBUG "758558720" 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG ] 12:39:15 DEBUG } 12:39:15 DEBUG } 12:39:21 INFO 12:39:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:39:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:39:21 INFO [loop_until]: OK (rc = 0) 12:39:21 DEBUG --- stdout --- 12:39:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 179m 1% 3859Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 143m 0% 19053Mi 66% 12:39:21 DEBUG --- stderr --- 12:39:21 DEBUG 12:39:25 INFO 12:39:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:39:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:39:25 INFO [loop_until]: OK (rc = 0) 12:39:25 DEBUG --- stdout --- 12:39:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 65m 13478Mi rcs-645c86fdd6-j4rsk 63m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:39:25 DEBUG --- stderr --- 12:39:25 DEBUG 12:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908014" 12:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908014" 12:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908014" 12:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908014" 12:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908014" 12:40:14 INFO [http_cmd]: http status code OK 12:40:14 DEBUG --- status code --- 12:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:14 DEBUG --- http response --- 12:40:14 INFO [http_cmd]: http status code OK 12:40:14 DEBUG --- status code --- 12:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:14 DEBUG --- http response --- 12:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:14 INFO [http_cmd]: http status code OK 12:40:14 ERROR [http_cmd]: ERROR 12:40:14 DEBUG --- status code --- 12:40:14 INFO [http_cmd]: http status code OK 12:40:14 ERROR --- status code --- 12:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:14 DEBUG --- status code --- 12:40:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:40:14 DEBUG --- http response --- 12:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:14 ERROR --- http response --- 12:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:14 DEBUG --- http response --- 12:40:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908014" 12:40:14 INFO [http_cmd]: http status code OK 12:40:14 DEBUG --- status code --- 12:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:14 DEBUG --- http response --- 12:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG { 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG "data": { 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG "result": [ 12:40:15 DEBUG { 12:40:15 DEBUG "metric": { 12:40:15 DEBUG "pod": "ds-cts-0" 12:40:15 DEBUG }, 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG [...] 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:40:15 DEBUG }, 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG "0" 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG { 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG "data": { 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG "result": [ 12:40:15 DEBUG { 12:40:15 DEBUG { 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG "metric": { 12:40:15 DEBUG "data": { 12:40:15 DEBUG "pod": "ds-cts-0" 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG }, 12:40:15 DEBUG "result": [ 12:40:15 DEBUG "value": [ 12:40:15 DEBUG { 12:40:15 DEBUG 12:40:15 DEBUG "metric": { 12:40:15 DEBUG 12:40:15 DEBUG "pod": "ds-cts-0" 12:40:15 DEBUG 12:40:15 DEBUG }, 12:40:15 DEBUG [...] 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:40:15 DEBUG [...] 12:40:15 DEBUG }, 12:40:15 DEBUG 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG 12:40:15 DEBUG "1038.533959148545" 12:40:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:40:15 DEBUG ] 12:40:15 DEBUG }, 12:40:15 DEBUG } 12:40:15 DEBUG "value": [ 12:40:15 DEBUG ] 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG } 12:40:15 DEBUG "0.16677933066564707" 12:40:15 DEBUG } 12:40:15 DEBUG ] 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908015" 12:40:15 DEBUG } 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG { 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG "data": { 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG "result": [ 12:40:15 DEBUG { 12:40:15 DEBUG "metric": { 12:40:15 DEBUG "instance": "10.142.0.11:9100" 12:40:15 DEBUG }, 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG [...] 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG "instance": "10.142.0.51:9100" 12:40:15 DEBUG }, 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG "0.00043333333333445503" 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG } 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908015" 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908015" 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG { 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG "status": "success", 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG "data": { 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG "result": [ 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG { 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 DEBUG "metric": { 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG "instance": "10.142.0.11:9100" 12:40:15 DEBUG { 12:40:15 DEBUG { 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG }, 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG "value": [ 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG "data": { 12:40:15 DEBUG "data": { 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG "result": [ 12:40:15 DEBUG "result": [ 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 DEBUG 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 DEBUG { 12:40:15 DEBUG { 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG [...] 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG "metric": { 12:40:15 DEBUG "metric": { 12:40:15 DEBUG { 12:40:15 DEBUG 12:40:15 DEBUG { 12:40:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:40:15 DEBUG "instance": "10.142.0.11:9100" 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG }, 12:40:15 DEBUG }, 12:40:15 DEBUG "data": { 12:40:15 DEBUG 12:40:15 DEBUG "data": { 12:40:15 DEBUG "value": [ 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 12:40:15 DEBUG "instance": "10.142.0.51:9100" 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG 12:40:15 DEBUG }, 12:40:15 DEBUG "result": [ 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908015" 12:40:15 DEBUG 12:40:15 DEBUG "result": [ 12:40:15 DEBUG 12:40:15 DEBUG "value": [ 12:40:15 DEBUG { 12:40:15 DEBUG 12:40:15 DEBUG { 12:40:15 DEBUG "metric": { 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG "metric": { 12:40:15 DEBUG [...] 12:40:15 DEBUG [...] 12:40:15 DEBUG "instance": "10.142.0.11:9100" 12:40:15 DEBUG "0.0006666666666670077" 12:40:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG }, 12:40:15 DEBUG ] 12:40:15 DEBUG }, 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG "value": [ 12:40:15 DEBUG } 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG ] 12:40:15 DEBUG 12:40:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:40:15 DEBUG "instance": "10.142.0.51:9100" 12:40:15 DEBUG 12:40:15 DEBUG } 12:40:15 DEBUG 12:40:15 DEBUG }, 12:40:15 DEBUG }, 12:40:15 DEBUG 12:40:15 DEBUG } 12:40:15 DEBUG 12:40:15 DEBUG "value": [ 12:40:15 DEBUG "value": [ 12:40:15 DEBUG [...] 12:40:15 DEBUG [...] 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908015" 12:40:15 DEBUG "0.029910395706318055" 12:40:15 DEBUG "1254208.0999999996" 12:40:15 DEBUG ] 12:40:15 DEBUG 12:40:15 DEBUG ] 12:40:15 DEBUG 12:40:15 DEBUG } 12:40:15 DEBUG 12:40:15 DEBUG } 12:40:15 DEBUG 12:40:15 DEBUG ] 12:40:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:40:15 DEBUG ] 12:40:15 DEBUG "instance": "10.142.0.51:9100" 12:40:15 DEBUG } 12:40:15 DEBUG }, 12:40:15 DEBUG } 12:40:15 DEBUG }, 12:40:15 DEBUG } 12:40:15 DEBUG "value": [ 12:40:15 DEBUG } 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG "14990020608" 12:40:15 DEBUG "19660.8" 12:40:15 DEBUG ] 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG } 12:40:15 DEBUG ] 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG } 12:40:15 DEBUG } 12:40:15 DEBUG } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG { 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG "data": { 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG "result": [ 12:40:15 DEBUG { 12:40:15 DEBUG "metric": { 12:40:15 DEBUG "pod": "ds-idrepo-0" 12:40:15 DEBUG }, 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG [...] 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:40:15 DEBUG }, 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG "758558720" 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG } 12:40:15 INFO [http_cmd]: http status code OK 12:40:15 DEBUG --- status code --- 12:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:40:15 DEBUG --- http response --- 12:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:40:15 DEBUG ----- output ----- 12:40:15 DEBUG { 12:40:15 DEBUG "status": "success", 12:40:15 DEBUG "data": { 12:40:15 DEBUG "resultType": "vector", 12:40:15 DEBUG "result": [ 12:40:15 DEBUG { 12:40:15 DEBUG "metric": { 12:40:15 DEBUG "instance": "10.142.0.11:9100" 12:40:15 DEBUG }, 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG [...] 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG 12:40:15 DEBUG "instance": "10.142.0.51:9100" 12:40:15 DEBUG }, 12:40:15 DEBUG "value": [ 12:40:15 DEBUG 1735908015, 12:40:15 DEBUG "1236885.533333333" 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG ] 12:40:15 DEBUG } 12:40:15 DEBUG } 12:40:21 INFO 12:40:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:40:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:40:21 INFO [loop_until]: OK (rc = 0) 12:40:21 DEBUG --- stdout --- 12:40:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 180m 1% 3871Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 147m 0% 19054Mi 66% 12:40:21 DEBUG --- stderr --- 12:40:21 DEBUG 12:40:25 INFO 12:40:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:40:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:40:25 INFO [loop_until]: OK (rc = 0) 12:40:25 DEBUG --- stdout --- 12:40:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 97m 13479Mi rcs-645c86fdd6-j4rsk 58m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:40:25 DEBUG --- stderr --- 12:40:25 DEBUG 12:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908074" 12:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908074" 12:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908074" 12:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908074" 12:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908074" 12:41:14 INFO [http_cmd]: http status code OK 12:41:14 INFO [http_cmd]: http status code OK 12:41:14 INFO [http_cmd]: http status code OK 12:41:14 ERROR [http_cmd]: ERROR 12:41:14 DEBUG --- status code --- 12:41:14 DEBUG --- status code --- 12:41:14 DEBUG --- status code --- 12:41:14 ERROR --- status code --- 12:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:41:14 DEBUG --- http response --- 12:41:14 DEBUG --- http response --- 12:41:14 DEBUG --- http response --- 12:41:14 ERROR --- http response --- 12:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:41:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:41:14 INFO [http_cmd]: http status code OK 12:41:14 DEBUG --- status code --- 12:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:14 DEBUG --- http response --- 12:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908074" 12:41:14 INFO [http_cmd]: http status code OK 12:41:14 DEBUG --- status code --- 12:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:14 DEBUG --- http response --- 12:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:41:15 DEBUG ----- output ----- 12:41:15 DEBUG { 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG "data": { 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG "result": [ 12:41:15 DEBUG { 12:41:15 DEBUG "metric": { 12:41:15 DEBUG "pod": "ds-cts-0" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG [...] 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG "pod": "ds-idrepo-0" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG "1267483.658095335" 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:41:15 DEBUG ----- output ----- 12:41:15 DEBUG { 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG "data": { 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG "result": [ 12:41:15 DEBUG { 12:41:15 DEBUG "metric": { 12:41:15 DEBUG "pod": "ds-cts-0" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG [...] 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG "pod": "ds-idrepo-0" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG "620.7069824169123" 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908075" 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:41:15 DEBUG ----- output ----- 12:41:15 DEBUG { 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG "data": { 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG "result": [ 12:41:15 DEBUG { 12:41:15 DEBUG "metric": { 12:41:15 DEBUG "pod": "ds-cts-0" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG [...] 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG "pod": "ds-idrepo-0" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG "0.04482046925476336" 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:41:15 DEBUG ----- output ----- 12:41:15 DEBUG { 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG "data": { 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG "result": [ 12:41:15 DEBUG { 12:41:15 DEBUG "metric": { 12:41:15 DEBUG "instance": "10.142.0.11:9100" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG [...] 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG "instance": "10.142.0.51:9100" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG "0.0006000000000047597" 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG } 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908075" 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908075" 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- http response --- 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908075" 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:41:15 DEBUG ----- output ----- 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG { 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:41:15 DEBUG "data": { 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG ----- output ----- 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG { 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:41:15 DEBUG "result": [ 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG ----- output ----- 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG { 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG "data": { 12:41:15 DEBUG { 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG "metric": { 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG "result": [ 12:41:15 DEBUG "data": { 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG }, 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:41:15 DEBUG { 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG "value": [ 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:41:15 DEBUG ----- output ----- 12:41:15 DEBUG "metric": { 12:41:15 DEBUG "result": [ 12:41:15 DEBUG 12:41:15 DEBUG ----- output ----- 12:41:15 DEBUG { 12:41:15 DEBUG "instance": "10.142.0.11:9100" 12:41:15 DEBUG { 12:41:15 DEBUG 12:41:15 DEBUG { 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG }, 12:41:15 DEBUG "metric": { 12:41:15 DEBUG 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG "data": { 12:41:15 DEBUG "value": [ 12:41:15 DEBUG "instance": "10.142.0.11:9100" 12:41:15 DEBUG [...] 12:41:15 DEBUG "data": { 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG 12:41:15 DEBUG }, 12:41:15 DEBUG 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG "result": [ 12:41:15 DEBUG 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 12:41:15 DEBUG "result": [ 12:41:15 DEBUG { 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG { 12:41:15 DEBUG "metric": { 12:41:15 DEBUG [...] 12:41:15 DEBUG 12:41:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:41:15 DEBUG "metric": { 12:41:15 DEBUG "instance": "10.142.0.11:9100" 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG }, 12:41:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:41:15 DEBUG }, 12:41:15 DEBUG 12:41:15 DEBUG [...] 12:41:15 DEBUG "value": [ 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 1735908075, 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 12:41:15 DEBUG "instance": "10.142.0.51:9100" 12:41:15 DEBUG 12:41:15 DEBUG "0.08910148263256588" 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG }, 12:41:15 DEBUG 12:41:15 DEBUG ] 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG "value": [ 12:41:15 DEBUG "instance": "10.142.0.51:9100" 12:41:15 DEBUG } 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG 12:41:15 DEBUG [...] 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG }, 12:41:15 DEBUG ] 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:41:15 DEBUG [...] 12:41:15 DEBUG 12:41:15 DEBUG "0.0003333333333330302" 12:41:15 DEBUG "value": [ 12:41:15 DEBUG } 12:41:15 DEBUG ----- output ----- 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG ] 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG } 12:41:15 DEBUG { 12:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908075" 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG } 12:41:15 DEBUG "1969513.4000000001" 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG 12:41:15 DEBUG "instance": "10.142.0.51:9100" 12:41:15 DEBUG ] 12:41:15 DEBUG ] 12:41:15 DEBUG "data": { 12:41:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:41:15 DEBUG }, 12:41:15 DEBUG } 12:41:15 DEBUG } 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG } 12:41:15 DEBUG ] 12:41:15 DEBUG "result": [ 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG } 12:41:15 DEBUG { 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG "27306.666666666664" 12:41:15 DEBUG } 12:41:15 DEBUG "metric": { 12:41:15 DEBUG "14995116032" 12:41:15 DEBUG ] 12:41:15 DEBUG "pod": "ds-idrepo-0" 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG }, 12:41:15 DEBUG } 12:41:15 DEBUG ] 12:41:15 DEBUG "value": [ 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG 12:41:15 DEBUG } 12:41:15 DEBUG } 12:41:15 DEBUG 12:41:15 DEBUG } 12:41:15 DEBUG 12:41:15 DEBUG [...] 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG "758558720" 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG } 12:41:15 INFO [http_cmd]: http status code OK 12:41:15 DEBUG --- status code --- 12:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:41:15 DEBUG --- http response --- 12:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:41:15 DEBUG ----- output ----- 12:41:15 DEBUG { 12:41:15 DEBUG "status": "success", 12:41:15 DEBUG "data": { 12:41:15 DEBUG "resultType": "vector", 12:41:15 DEBUG "result": [ 12:41:15 DEBUG { 12:41:15 DEBUG "metric": { 12:41:15 DEBUG "instance": "10.142.0.11:9100" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG [...] 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG 12:41:15 DEBUG "instance": "10.142.0.51:9100" 12:41:15 DEBUG }, 12:41:15 DEBUG "value": [ 12:41:15 DEBUG 1735908075, 12:41:15 DEBUG "1952570.7333333327" 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG ] 12:41:15 DEBUG } 12:41:15 DEBUG } 12:41:21 INFO 12:41:21 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:41:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:41:22 INFO [loop_until]: OK (rc = 0) 12:41:22 DEBUG --- stdout --- 12:41:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 201m 1% 3866Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 185m 0% 19055Mi 67% 12:41:22 DEBUG --- stderr --- 12:41:22 DEBUG 12:41:25 INFO 12:41:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:41:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:41:25 INFO [loop_until]: OK (rc = 0) 12:41:25 DEBUG --- stdout --- 12:41:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 84m 13479Mi rcs-645c86fdd6-j4rsk 109m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:41:25 DEBUG --- stderr --- 12:41:25 DEBUG 12:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908134" 12:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908134" 12:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908134" 12:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908134" 12:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908134" 12:42:14 INFO [http_cmd]: http status code OK 12:42:14 DEBUG --- status code --- 12:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:14 DEBUG --- http response --- 12:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:14 INFO [http_cmd]: http status code OK 12:42:14 DEBUG --- status code --- 12:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:14 DEBUG --- http response --- 12:42:14 INFO [http_cmd]: http status code OK 12:42:14 DEBUG --- status code --- 12:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:14 DEBUG --- http response --- 12:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:14 ERROR [http_cmd]: ERROR 12:42:14 ERROR --- status code --- 12:42:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:42:14 ERROR --- http response --- 12:42:14 INFO [http_cmd]: http status code OK 12:42:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:42:14 DEBUG --- status code --- 12:42:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:14 DEBUG --- http response --- 12:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908134" 12:42:14 INFO [http_cmd]: http status code OK 12:42:14 DEBUG --- status code --- 12:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:14 DEBUG --- http response --- 12:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG --- status code --- 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG { 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG "data": { 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "result": [ 12:42:15 DEBUG { 12:42:15 DEBUG "metric": { 12:42:15 DEBUG "pod": "ds-cts-0" 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG [...] 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG "0" 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG { 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG "data": { 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "result": [ 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG { 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG "metric": { 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG "pod": "ds-cts-0" 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG }, 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG "value": [ 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG 12:42:15 DEBUG { 12:42:15 DEBUG 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG 12:42:15 DEBUG "data": { 12:42:15 DEBUG [...] 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG 12:42:15 DEBUG "result": [ 12:42:15 DEBUG 12:42:15 DEBUG { 12:42:15 DEBUG 12:42:15 DEBUG "metric": { 12:42:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:42:15 DEBUG "pod": "ds-cts-0" 12:42:15 DEBUG }, 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG 12:42:15 DEBUG "258.87185969347445" 12:42:15 DEBUG 12:42:15 DEBUG ] 12:42:15 DEBUG 12:42:15 DEBUG } 12:42:15 DEBUG [...] 12:42:15 DEBUG ] 12:42:15 DEBUG 12:42:15 DEBUG } 12:42:15 DEBUG 12:42:15 DEBUG } 12:42:15 DEBUG 12:42:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG "0.0005959551271920355" 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG { 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG "data": { 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "result": [ 12:42:15 DEBUG { 12:42:15 DEBUG "metric": { 12:42:15 DEBUG "instance": "10.142.0.11:9100" 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG [...] 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG "instance": "10.142.0.51:9100" 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG "0.0011999999999981507" 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908135" 12:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908135" 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG { 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG { 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG { 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG "data": { 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG "data": { 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "data": { 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "result": [ 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "result": [ 12:42:15 DEBUG { 12:42:15 DEBUG "result": [ 12:42:15 DEBUG { 12:42:15 DEBUG "metric": { 12:42:15 DEBUG { 12:42:15 DEBUG "metric": { 12:42:15 DEBUG "instance": "10.142.0.11:9100" 12:42:15 DEBUG "metric": { 12:42:15 DEBUG "instance": "10.142.0.11:9100" 12:42:15 DEBUG }, 12:42:15 DEBUG "instance": "10.142.0.11:9100" 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG [...] 12:42:15 DEBUG 12:42:15 DEBUG [...] 12:42:15 DEBUG 12:42:15 DEBUG [...] 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG "instance": "10.142.0.51:9100" 12:42:15 DEBUG 12:42:15 DEBUG "instance": "10.142.0.51:9100" 12:42:15 DEBUG }, 12:42:15 DEBUG "instance": "10.142.0.51:9100" 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG "2389155.4000000004" 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG "0.0009999999999990906" 12:42:15 DEBUG ] 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG "31539.199999999997" 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 DEBUG ] 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG } 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG } 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG { 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG "data": { 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "result": [ 12:42:15 DEBUG { 12:42:15 DEBUG "metric": { 12:42:15 DEBUG "instance": "10.142.0.11:9100" 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG [...] 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG "instance": "10.142.0.51:9100" 12:42:15 DEBUG }, 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG "2372216.1666666674" 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- http response --- 12:42:15 INFO [http_cmd]: http status code OK 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG --- status code --- 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG --- http response --- 12:42:15 DEBUG { 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG ----- output ----- 12:42:15 DEBUG "data": { 12:42:15 DEBUG { 12:42:15 DEBUG { 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG "status": "success", 12:42:15 DEBUG "result": [ 12:42:15 DEBUG "data": { 12:42:15 DEBUG "data": { 12:42:15 DEBUG { 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "resultType": "vector", 12:42:15 DEBUG "metric": { 12:42:15 DEBUG "result": [ 12:42:15 DEBUG "result": [ 12:42:15 DEBUG "pod": "ds-idrepo-0" 12:42:15 DEBUG { 12:42:15 DEBUG { 12:42:15 DEBUG }, 12:42:15 DEBUG "metric": { 12:42:15 DEBUG "metric": { 12:42:15 DEBUG "value": [ 12:42:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:42:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:42:15 DEBUG 12:42:15 DEBUG }, 12:42:15 DEBUG }, 12:42:15 DEBUG 12:42:15 DEBUG "value": [ 12:42:15 DEBUG "value": [ 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG [...] 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG [...] 12:42:15 DEBUG [...] 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG }, 12:42:15 DEBUG 12:42:15 DEBUG 12:42:15 DEBUG "value": [ 12:42:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:42:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG }, 12:42:15 DEBUG }, 12:42:15 DEBUG "758558720" 12:42:15 DEBUG "value": [ 12:42:15 DEBUG "value": [ 12:42:15 DEBUG ] 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG 1735908135, 12:42:15 DEBUG } 12:42:15 DEBUG "14990016512" 12:42:15 DEBUG "0.130376121518553" 12:42:15 DEBUG ] 12:42:15 DEBUG ] 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 DEBUG ] 12:42:15 DEBUG ] 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:15 DEBUG } 12:42:22 INFO 12:42:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:42:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:42:22 INFO [loop_until]: OK (rc = 0) 12:42:22 DEBUG --- stdout --- 12:42:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 238m 1% 3866Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 181m 0% 19058Mi 67% 12:42:22 DEBUG --- stderr --- 12:42:22 DEBUG 12:42:25 INFO 12:42:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:42:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:42:25 INFO [loop_until]: OK (rc = 0) 12:42:25 DEBUG --- stdout --- 12:42:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 102m 13479Mi rcs-645c86fdd6-j4rsk 103m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:42:25 DEBUG --- stderr --- 12:42:25 DEBUG 12:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908194" 12:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908194" 12:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908194" 12:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908194" 12:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908194" 12:43:14 INFO [http_cmd]: http status code OK 12:43:14 DEBUG --- status code --- 12:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:14 DEBUG --- http response --- 12:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:14 ERROR [http_cmd]: ERROR 12:43:14 ERROR --- status code --- 12:43:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:43:14 ERROR --- http response --- 12:43:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:43:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:43:14 INFO [http_cmd]: http status code OK 12:43:14 DEBUG --- status code --- 12:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:14 DEBUG --- http response --- 12:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:14 INFO [http_cmd]: http status code OK 12:43:14 DEBUG --- status code --- 12:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:14 DEBUG --- http response --- 12:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:14 INFO [http_cmd]: http status code OK 12:43:14 DEBUG --- status code --- 12:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:14 DEBUG --- http response --- 12:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908194" 12:43:14 INFO [http_cmd]: http status code OK 12:43:14 DEBUG --- status code --- 12:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:14 DEBUG --- http response --- 12:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG { 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG "data": { 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG "result": [ 12:43:15 DEBUG { 12:43:15 DEBUG "metric": { 12:43:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:43:15 DEBUG }, 12:43:15 DEBUG "value": [ 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG [...] 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG "pod": "ds-cts-0" 12:43:15 DEBUG }, 12:43:15 DEBUG "value": [ 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG "0" 12:43:15 DEBUG ] 12:43:15 DEBUG } 12:43:15 DEBUG ] 12:43:15 DEBUG } 12:43:15 DEBUG } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG { 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG "data": { 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG "result": [ 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG { 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG "metric": { 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG }, 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:43:15 DEBUG "value": [ 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG 12:43:15 DEBUG { 12:43:15 DEBUG { 12:43:15 DEBUG 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG 12:43:15 DEBUG "data": { 12:43:15 DEBUG "data": { 12:43:15 DEBUG [...] 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG 12:43:15 DEBUG "result": [ 12:43:15 DEBUG "result": [ 12:43:15 DEBUG 12:43:15 DEBUG { 12:43:15 DEBUG { 12:43:15 DEBUG 12:43:15 DEBUG "metric": { 12:43:15 DEBUG "metric": { 12:43:15 DEBUG "pod": "ds-cts-0" 12:43:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:43:15 DEBUG "instance": "10.142.0.11:9100" 12:43:15 DEBUG }, 12:43:15 DEBUG }, 12:43:15 DEBUG }, 12:43:15 DEBUG "value": [ 12:43:15 DEBUG "value": [ 12:43:15 DEBUG "value": [ 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG "422.0890331464542" 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG ] 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG } 12:43:15 DEBUG [...] 12:43:15 DEBUG [...] 12:43:15 DEBUG ] 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG } 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG } 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG "pod": "ds-cts-0" 12:43:15 DEBUG "instance": "10.142.0.51:9100" 12:43:15 DEBUG }, 12:43:15 DEBUG }, 12:43:15 DEBUG "value": [ 12:43:15 DEBUG "value": [ 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG "0.001955175963907416" 12:43:15 DEBUG "0.00046666666666700016" 12:43:15 DEBUG ] 12:43:15 DEBUG ] 12:43:15 DEBUG } 12:43:15 DEBUG } 12:43:15 DEBUG ] 12:43:15 DEBUG ] 12:43:15 DEBUG } 12:43:15 DEBUG } 12:43:15 DEBUG } 12:43:15 DEBUG } 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908195" 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG { 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG "data": { 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG "result": [ 12:43:15 DEBUG { 12:43:15 DEBUG "metric": { 12:43:15 DEBUG "instance": "10.142.0.11:9100" 12:43:15 DEBUG }, 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908195" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG "value": [ 12:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908195" 12:43:15 DEBUG --- status code --- 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:43:15 DEBUG [...] 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG 12:43:15 DEBUG { 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG { 12:43:15 DEBUG 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG { 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG 12:43:15 DEBUG "data": { 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG "data": { 12:43:15 DEBUG "instance": "10.142.0.51:9100" 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG "data": { 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG }, 12:43:15 DEBUG "result": [ 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG "result": [ 12:43:15 DEBUG "value": [ 12:43:15 DEBUG { 12:43:15 DEBUG "result": [ 12:43:15 DEBUG { 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG "metric": { 12:43:15 DEBUG { 12:43:15 DEBUG "metric": { 12:43:15 DEBUG "2365888.5333333327" 12:43:15 DEBUG "instance": "10.142.0.11:9100" 12:43:15 DEBUG "metric": { 12:43:15 DEBUG "instance": "10.142.0.11:9100" 12:43:15 DEBUG ] 12:43:15 DEBUG }, 12:43:15 DEBUG "instance": "10.142.0.11:9100" 12:43:15 DEBUG }, 12:43:15 DEBUG } 12:43:15 DEBUG "value": [ 12:43:15 DEBUG }, 12:43:15 DEBUG "value": [ 12:43:15 DEBUG ] 12:43:15 DEBUG 12:43:15 DEBUG "value": [ 12:43:15 DEBUG 12:43:15 DEBUG } 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG } 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG [...] 12:43:15 DEBUG 12:43:15 DEBUG [...] 12:43:15 DEBUG 12:43:15 DEBUG [...] 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG "instance": "10.142.0.51:9100" 12:43:15 DEBUG 12:43:15 DEBUG "instance": "10.142.0.51:9100" 12:43:15 DEBUG }, 12:43:15 DEBUG "instance": "10.142.0.51:9100" 12:43:15 DEBUG }, 12:43:15 DEBUG "value": [ 12:43:15 DEBUG }, 12:43:15 DEBUG "value": [ 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG "value": [ 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG "24439.466666666667" 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG "2383262.1000000015" 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG ] 12:43:15 INFO [http_cmd]: http status code OK 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG } 12:43:15 DEBUG "0.0006666666666674814" 12:43:15 DEBUG ] 12:43:15 DEBUG --- status code --- 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG ] 12:43:15 DEBUG ] 12:43:15 DEBUG } 12:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG } 12:43:15 DEBUG } 12:43:15 DEBUG ] 12:43:15 DEBUG --- http response --- 12:43:15 DEBUG { 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:43:15 DEBUG } 12:43:15 DEBUG ] 12:43:15 DEBUG } 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG { 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG "data": { 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG "result": [ 12:43:15 DEBUG } 12:43:15 DEBUG "data": { 12:43:15 DEBUG ----- output ----- 12:43:15 DEBUG } 12:43:15 DEBUG { 12:43:15 DEBUG } 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG { 12:43:15 DEBUG "metric": { 12:43:15 DEBUG "result": [ 12:43:15 DEBUG "status": "success", 12:43:15 DEBUG "pod": "ds-idrepo-0" 12:43:15 DEBUG { 12:43:15 DEBUG "data": { 12:43:15 DEBUG }, 12:43:15 DEBUG "metric": { 12:43:15 DEBUG "resultType": "vector", 12:43:15 DEBUG "value": [ 12:43:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:43:15 DEBUG "result": [ 12:43:15 DEBUG 12:43:15 DEBUG }, 12:43:15 DEBUG { 12:43:15 DEBUG 12:43:15 DEBUG "value": [ 12:43:15 DEBUG "metric": { 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:43:15 DEBUG [...] 12:43:15 DEBUG 12:43:15 DEBUG }, 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG "value": [ 12:43:15 DEBUG 12:43:15 DEBUG [...] 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:43:15 DEBUG 12:43:15 DEBUG 12:43:15 DEBUG }, 12:43:15 DEBUG 12:43:15 DEBUG [...] 12:43:15 DEBUG "value": [ 12:43:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:43:15 DEBUG 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG }, 12:43:15 DEBUG 12:43:15 DEBUG "758558720" 12:43:15 DEBUG "value": [ 12:43:15 DEBUG 12:43:15 DEBUG ] 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:43:15 DEBUG } 12:43:15 DEBUG "14988783616" 12:43:15 DEBUG }, 12:43:15 DEBUG ] 12:43:15 DEBUG ] 12:43:15 DEBUG "value": [ 12:43:15 DEBUG } 12:43:15 DEBUG } 12:43:15 DEBUG 1735908195, 12:43:15 DEBUG } 12:43:15 DEBUG ] 12:43:15 DEBUG "0.10080289506605801" 12:43:15 DEBUG } 12:43:15 DEBUG ] 12:43:15 DEBUG } 12:43:15 DEBUG } 12:43:15 DEBUG ] 12:43:15 DEBUG } 12:43:15 DEBUG } 12:43:22 INFO 12:43:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:43:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:43:22 INFO [loop_until]: OK (rc = 0) 12:43:22 DEBUG --- stdout --- 12:43:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 236m 1% 3874Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 752m 2% 19053Mi 66% 12:43:22 DEBUG --- stderr --- 12:43:22 DEBUG 12:43:25 INFO 12:43:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:43:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:43:25 INFO [loop_until]: OK (rc = 0) 12:43:25 DEBUG --- stdout --- 12:43:25 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 372m 13480Mi rcs-645c86fdd6-j4rsk 106m 823Mi rcs-645c86fdd6-m29zn 2m 361Mi 12:43:25 DEBUG --- stderr --- 12:43:25 DEBUG 12:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908254" 12:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908254" 12:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908254" 12:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908254" 12:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908254" 12:44:14 INFO [http_cmd]: http status code OK 12:44:14 DEBUG --- status code --- 12:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:14 DEBUG --- http response --- 12:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:14 INFO [http_cmd]: http status code OK 12:44:14 DEBUG --- status code --- 12:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:14 DEBUG --- http response --- 12:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:14 ERROR [http_cmd]: ERROR 12:44:14 ERROR --- status code --- 12:44:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:44:14 ERROR --- http response --- 12:44:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:44:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:44:14 INFO [http_cmd]: http status code OK 12:44:14 INFO [http_cmd]: http status code OK 12:44:14 DEBUG --- status code --- 12:44:14 DEBUG --- status code --- 12:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:14 DEBUG --- http response --- 12:44:14 DEBUG --- http response --- 12:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908254" 12:44:14 INFO [http_cmd]: http status code OK 12:44:14 DEBUG --- status code --- 12:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:14 DEBUG --- http response --- 12:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735908255, "0" ] } ] } } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735908255, "0" ] } ] } } 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735908255, "0.0006116298120985446" ] } ] } } 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:44:15 DEBUG ----- output ----- 12:44:15 DEBUG { 12:44:15 DEBUG "status": "success", 12:44:15 DEBUG "data": { 12:44:15 DEBUG "resultType": "vector", 12:44:15 DEBUG "result": [ 12:44:15 DEBUG { 12:44:15 DEBUG "metric": { 12:44:15 DEBUG "instance": "10.142.0.11:9100" 12:44:15 DEBUG }, 12:44:15 DEBUG "value": [ 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG [...] 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG "instance": "10.142.0.51:9100" 12:44:15 DEBUG }, 12:44:15 DEBUG "value": [ 12:44:15 DEBUG 1735908255, 12:44:15 DEBUG "0.0002666666666717295" 12:44:15 DEBUG ] 12:44:15 DEBUG } 12:44:15 DEBUG ] 12:44:15 DEBUG } 12:44:15 DEBUG } 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908255" 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908255" 12:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908255" 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:44:15 DEBUG ----- output ----- 12:44:15 DEBUG { 12:44:15 DEBUG "status": "success", 12:44:15 DEBUG "data": { 12:44:15 DEBUG "resultType": "vector", 12:44:15 DEBUG "result": [ 12:44:15 DEBUG { 12:44:15 DEBUG "metric": { 12:44:15 DEBUG "instance": "10.142.0.11:9100" 12:44:15 DEBUG }, 12:44:15 DEBUG "value": [ 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG [...] 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG "instance": "10.142.0.51:9100" 12:44:15 DEBUG }, 12:44:15 DEBUG "value": [ 12:44:15 DEBUG 1735908255, 12:44:15 DEBUG "1421489.1999999995" 12:44:15 DEBUG ] 12:44:15 DEBUG } 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG ] 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG } 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG } 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:44:15 DEBUG ----- output ----- 12:44:15 DEBUG ----- output ----- 12:44:15 DEBUG ----- output ----- 12:44:15 DEBUG { 12:44:15 DEBUG { 12:44:15 DEBUG { 12:44:15 DEBUG "status": "success", 12:44:15 DEBUG "status": "success", 12:44:15 DEBUG "status": "success", 12:44:15 DEBUG "data": { 12:44:15 DEBUG "data": { 12:44:15 DEBUG "data": { 12:44:15 DEBUG "resultType": "vector", 12:44:15 DEBUG "resultType": "vector", 12:44:15 DEBUG "resultType": "vector", 12:44:15 DEBUG "result": [ 12:44:15 DEBUG "result": [ 12:44:15 DEBUG "result": [ 12:44:15 DEBUG { 12:44:15 DEBUG { 12:44:15 DEBUG { 12:44:15 DEBUG "metric": { 12:44:15 DEBUG "metric": { 12:44:15 DEBUG "metric": { 12:44:15 DEBUG "instance": "10.142.0.11:9100" 12:44:15 DEBUG }, 12:44:15 DEBUG "instance": "10.142.0.11:9100" 12:44:15 DEBUG "instance": "10.142.0.11:9100" 12:44:15 DEBUG "value": [ 12:44:15 DEBUG }, 12:44:15 DEBUG }, 12:44:15 DEBUG 12:44:15 DEBUG "value": [ 12:44:15 DEBUG "value": [ 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG [...] 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG [...] 12:44:15 DEBUG [...] 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG "instance": "10.142.0.51:9100" 12:44:15 DEBUG }, 12:44:15 DEBUG "instance": "10.142.0.51:9100" 12:44:15 DEBUG 12:44:15 DEBUG "value": [ 12:44:15 DEBUG }, 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG 12:44:15 DEBUG 1735908255, 12:44:15 DEBUG "value": [ 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 INFO [http_cmd]: http status code OK 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG "instance": "10.142.0.51:9100" 12:44:15 DEBUG "1438543.6333333333" 12:44:15 DEBUG 1735908255, 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG --- status code --- 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG }, 12:44:15 DEBUG ] 12:44:15 DEBUG "7782.400000000001" 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG "value": [ 12:44:15 DEBUG 1735908255, 12:44:15 DEBUG "0" 12:44:15 DEBUG ] 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:44:15 DEBUG } 12:44:15 DEBUG ] 12:44:15 DEBUG --- http response --- 12:44:15 DEBUG } 12:44:15 DEBUG ----- output ----- 12:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:44:15 DEBUG ] 12:44:15 DEBUG } 12:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:44:15 DEBUG ] 12:44:15 DEBUG { 12:44:15 DEBUG ----- output ----- 12:44:15 DEBUG } 12:44:15 DEBUG ] 12:44:15 DEBUG ----- output ----- 12:44:15 DEBUG } 12:44:15 DEBUG "status": "success", 12:44:15 DEBUG { 12:44:15 DEBUG "status": "success", 12:44:15 DEBUG "data": { 12:44:15 DEBUG "resultType": "vector", 12:44:15 DEBUG "result": [ 12:44:15 DEBUG "data": { 12:44:15 DEBUG } 12:44:15 DEBUG } 12:44:15 DEBUG { 12:44:15 DEBUG } 12:44:15 DEBUG { 12:44:15 DEBUG "resultType": "vector", 12:44:15 DEBUG } 12:44:15 DEBUG "status": "success", 12:44:15 DEBUG "metric": { 12:44:15 DEBUG "result": [ 12:44:15 DEBUG "data": { 12:44:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:44:15 DEBUG { 12:44:15 DEBUG "resultType": "vector", 12:44:15 DEBUG }, 12:44:15 DEBUG "metric": { 12:44:15 DEBUG "result": [ 12:44:15 DEBUG "value": [ 12:44:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:44:15 DEBUG { 12:44:15 DEBUG 12:44:15 DEBUG }, 12:44:15 DEBUG "metric": { 12:44:15 DEBUG 12:44:15 DEBUG "value": [ 12:44:15 DEBUG "pod": "ds-idrepo-0" 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG }, 12:44:15 DEBUG [...] 12:44:15 DEBUG 12:44:15 DEBUG "value": [ 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG [...] 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:44:15 DEBUG 12:44:15 DEBUG [...] 12:44:15 DEBUG }, 12:44:15 DEBUG 12:44:15 DEBUG 12:44:15 DEBUG "value": [ 12:44:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:44:15 DEBUG 12:44:15 DEBUG 1735908255, 12:44:15 DEBUG }, 12:44:15 DEBUG 12:44:15 DEBUG "0.7347606078579376" 12:44:15 DEBUG "value": [ 12:44:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:44:15 DEBUG ] 12:44:15 DEBUG 1735908255, 12:44:15 DEBUG }, 12:44:15 DEBUG } 12:44:15 DEBUG "14986481664" 12:44:15 DEBUG "value": [ 12:44:15 DEBUG ] 12:44:15 DEBUG ] 12:44:15 DEBUG 1735908255, 12:44:15 DEBUG } 12:44:15 DEBUG } 12:44:15 DEBUG "758960128" 12:44:15 DEBUG } 12:44:15 DEBUG ] 12:44:15 DEBUG ] 12:44:15 DEBUG } 12:44:15 DEBUG } 12:44:15 DEBUG } 12:44:15 DEBUG ] 12:44:15 DEBUG } 12:44:15 DEBUG } 12:44:22 INFO 12:44:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:44:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:44:22 INFO [loop_until]: OK (rc = 0) 12:44:22 DEBUG --- stdout --- 12:44:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 205m 1% 3884Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 172m 0% 19053Mi 66% 12:44:22 DEBUG --- stderr --- 12:44:22 DEBUG 12:44:25 INFO 12:44:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:44:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:44:26 INFO [loop_until]: OK (rc = 0) 12:44:26 DEBUG --- stdout --- 12:44:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 61m 13479Mi rcs-645c86fdd6-j4rsk 104m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:44:26 DEBUG --- stderr --- 12:44:26 DEBUG 12:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908314" 12:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908314" 12:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908314" 12:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908314" 12:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908314" 12:45:14 ERROR [http_cmd]: ERROR 12:45:14 ERROR --- status code --- 12:45:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:45:14 ERROR --- http response --- 12:45:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:45:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:45:14 INFO [http_cmd]: http status code OK 12:45:14 DEBUG --- status code --- 12:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:14 DEBUG --- http response --- 12:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:14 INFO [http_cmd]: http status code OK 12:45:14 DEBUG --- status code --- 12:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:14 DEBUG --- http response --- 12:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:14 INFO [http_cmd]: http status code OK 12:45:14 DEBUG --- status code --- 12:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:14 INFO [http_cmd]: http status code OK 12:45:14 DEBUG --- http response --- 12:45:14 DEBUG --- status code --- 12:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:14 DEBUG --- http response --- 12:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908314" 12:45:14 INFO [http_cmd]: http status code OK 12:45:14 DEBUG --- status code --- 12:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:14 DEBUG --- http response --- 12:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG { 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG "data": { 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "result": [ 12:45:15 DEBUG { 12:45:15 DEBUG "metric": { 12:45:15 DEBUG "pod": "ds-cts-0" 12:45:15 DEBUG }, 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG [...] 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:45:15 DEBUG }, 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG "0" 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG { 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG "data": { 12:45:15 DEBUG { 12:45:15 DEBUG { 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG "result": [ 12:45:15 DEBUG "data": { 12:45:15 DEBUG "data": { 12:45:15 DEBUG { 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "metric": { 12:45:15 DEBUG "result": [ 12:45:15 DEBUG "result": [ 12:45:15 DEBUG "instance": "10.142.0.11:9100" 12:45:15 DEBUG { 12:45:15 DEBUG { 12:45:15 DEBUG }, 12:45:15 DEBUG "metric": { 12:45:15 DEBUG "metric": { 12:45:15 DEBUG "value": [ 12:45:15 DEBUG "pod": "ds-cts-0" 12:45:15 DEBUG "pod": "ds-cts-0" 12:45:15 DEBUG 12:45:15 DEBUG }, 12:45:15 DEBUG }, 12:45:15 DEBUG 12:45:15 DEBUG "value": [ 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG [...] 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG [...] 12:45:15 DEBUG [...] 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG "instance": "10.142.0.51:9100" 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG }, 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG "value": [ 12:45:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:45:15 DEBUG "pod": "ds-idrepo-0" 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG }, 12:45:15 DEBUG }, 12:45:15 DEBUG "0.0003999999999981204" 12:45:15 DEBUG "value": [ 12:45:15 DEBUG "value": [ 12:45:15 DEBUG ] 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG } 12:45:15 DEBUG "0" 12:45:15 DEBUG "0" 12:45:15 DEBUG ] 12:45:15 DEBUG ] 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 DEBUG ] 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908315" 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908315" 12:45:15 DEBUG --- status code --- 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908315" 12:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908315" 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG { 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG "data": { 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "result": [ 12:45:15 DEBUG { 12:45:15 DEBUG "metric": { 12:45:15 DEBUG "instance": "10.142.0.11:9100" 12:45:15 DEBUG }, 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG [...] 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG "instance": "10.142.0.51:9100" 12:45:15 DEBUG }, 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG "1225148.6333333333" 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG { 12:45:15 DEBUG { 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG "data": { 12:45:15 DEBUG { 12:45:15 DEBUG { 12:45:15 DEBUG "status": "success", 12:45:15 INFO [http_cmd]: http status code OK 12:45:15 DEBUG { 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG "data": { 12:45:15 DEBUG --- status code --- 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG "result": [ 12:45:15 DEBUG "data": { 12:45:15 DEBUG "data": { 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:15 DEBUG "data": { 12:45:15 DEBUG { 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "result": [ 12:45:15 DEBUG --- http response --- 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "metric": { 12:45:15 DEBUG "result": [ 12:45:15 DEBUG "result": [ 12:45:15 DEBUG { 12:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:45:15 DEBUG "result": [ 12:45:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:45:15 DEBUG { 12:45:15 DEBUG { 12:45:15 DEBUG "metric": { 12:45:15 DEBUG ----- output ----- 12:45:15 DEBUG { 12:45:15 DEBUG }, 12:45:15 DEBUG "metric": { 12:45:15 DEBUG "metric": { 12:45:15 DEBUG "instance": "10.142.0.11:9100" 12:45:15 DEBUG { 12:45:15 DEBUG "metric": { 12:45:15 DEBUG "value": [ 12:45:15 DEBUG "instance": "10.142.0.11:9100" 12:45:15 DEBUG "instance": "10.142.0.11:9100" 12:45:15 DEBUG }, 12:45:15 DEBUG "status": "success", 12:45:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:45:15 DEBUG 12:45:15 DEBUG }, 12:45:15 DEBUG }, 12:45:15 DEBUG "value": [ 12:45:15 DEBUG "data": { 12:45:15 DEBUG }, 12:45:15 DEBUG 12:45:15 DEBUG "value": [ 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 12:45:15 DEBUG "resultType": "vector", 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG "result": [ 12:45:15 DEBUG 12:45:15 DEBUG [...] 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG { 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG [...] 12:45:15 DEBUG "metric": { 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG [...] 12:45:15 DEBUG [...] 12:45:15 DEBUG 12:45:15 DEBUG "pod": "ds-idrepo-0" 12:45:15 DEBUG [...] 12:45:15 DEBUG 12:45:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG }, 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG }, 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 12:45:15 DEBUG "instance": "10.142.0.51:9100" 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG "instance": "10.142.0.51:9100" 12:45:15 DEBUG }, 12:45:15 DEBUG 12:45:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:45:15 DEBUG "instance": "10.142.0.51:9100" 12:45:15 DEBUG "0.14524857534422064" 12:45:15 DEBUG }, 12:45:15 DEBUG "value": [ 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG 12:45:15 DEBUG }, 12:45:15 DEBUG ] 12:45:15 DEBUG "value": [ 12:45:15 DEBUG "25804.8" 12:45:15 DEBUG }, 12:45:15 DEBUG [...] 12:45:15 DEBUG "value": [ 12:45:15 DEBUG } 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG ] 12:45:15 DEBUG "1241975.0333333334" 12:45:15 DEBUG "value": [ 12:45:15 DEBUG ] 12:45:15 DEBUG 12:45:15 DEBUG "0" 12:45:15 DEBUG } 12:45:15 DEBUG ] 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG } 12:45:15 DEBUG 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 DEBUG "14990868480" 12:45:15 DEBUG } 12:45:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:45:15 DEBUG } 12:45:15 DEBUG ] 12:45:15 DEBUG ] 12:45:15 DEBUG }, 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 DEBUG "value": [ 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 DEBUG ] 12:45:15 DEBUG 1735908315, 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:15 DEBUG "758771712" 12:45:15 DEBUG } 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG ] 12:45:15 DEBUG } 12:45:15 DEBUG } 12:45:22 INFO 12:45:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:45:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:45:22 INFO [loop_until]: OK (rc = 0) 12:45:22 DEBUG --- stdout --- 12:45:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 174m 1% 3882Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 157m 0% 19060Mi 67% 12:45:22 DEBUG --- stderr --- 12:45:22 DEBUG 12:45:26 INFO 12:45:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:45:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:45:26 INFO [loop_until]: OK (rc = 0) 12:45:26 DEBUG --- stdout --- 12:45:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 86m 13479Mi rcs-645c86fdd6-j4rsk 56m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:45:26 DEBUG --- stderr --- 12:45:26 DEBUG 12:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908374" 12:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908374" 12:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908374" 12:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908374" 12:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908374" 12:46:14 INFO [http_cmd]: http status code OK 12:46:14 DEBUG --- status code --- 12:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:14 DEBUG --- http response --- 12:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:14 INFO [http_cmd]: http status code OK 12:46:14 INFO [http_cmd]: http status code OK 12:46:14 ERROR [http_cmd]: ERROR 12:46:14 DEBUG --- status code --- 12:46:14 DEBUG --- status code --- 12:46:14 ERROR --- status code --- 12:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:46:14 DEBUG --- http response --- 12:46:14 DEBUG --- http response --- 12:46:14 ERROR --- http response --- 12:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:14 INFO [http_cmd]: http status code OK 12:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:46:14 DEBUG --- status code --- 12:46:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:14 DEBUG --- http response --- 12:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908374" 12:46:14 INFO [http_cmd]: http status code OK 12:46:14 DEBUG --- status code --- 12:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:14 DEBUG --- http response --- 12:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG { 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG "data": { 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG "result": [ 12:46:15 DEBUG { 12:46:15 DEBUG "metric": { 12:46:15 DEBUG "pod": "ds-cts-0" 12:46:15 DEBUG }, 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG [...] 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:46:15 DEBUG }, 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG "0" 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:15 DEBUG { 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG { 12:46:15 DEBUG "data": { 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG "data": { 12:46:15 DEBUG "result": [ 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG { 12:46:15 DEBUG "result": [ 12:46:15 DEBUG "metric": { 12:46:15 DEBUG { 12:46:15 DEBUG "instance": "10.142.0.11:9100" 12:46:15 DEBUG "metric": { 12:46:15 DEBUG }, 12:46:15 DEBUG "pod": "ds-cts-0" 12:46:15 DEBUG "value": [ 12:46:15 DEBUG }, 12:46:15 DEBUG 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG 12:46:15 DEBUG [...] 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG [...] 12:46:15 DEBUG 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:46:15 DEBUG 12:46:15 DEBUG "instance": "10.142.0.51:9100" 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG 12:46:15 DEBUG }, 12:46:15 DEBUG { 12:46:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:46:15 DEBUG "value": [ 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG }, 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG "data": { 12:46:15 DEBUG "value": [ 12:46:15 DEBUG "0.00046666666666794754" 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG ] 12:46:15 DEBUG "result": [ 12:46:15 DEBUG "0" 12:46:15 DEBUG } 12:46:15 DEBUG { 12:46:15 DEBUG ] 12:46:15 DEBUG ] 12:46:15 DEBUG "metric": { 12:46:15 DEBUG } 12:46:15 DEBUG } 12:46:15 DEBUG "pod": "ds-cts-0" 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG }, 12:46:15 DEBUG } 12:46:15 DEBUG "value": [ 12:46:15 DEBUG } 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG [...] 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG "pod": "ds-idrepo-0" 12:46:15 DEBUG }, 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG "0" 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG { 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG "data": { 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG "result": [ 12:46:15 DEBUG { 12:46:15 DEBUG "metric": { 12:46:15 DEBUG "instance": "10.142.0.11:9100" 12:46:15 DEBUG }, 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG [...] 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG "instance": "10.142.0.51:9100" 12:46:15 DEBUG }, 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG "1124022.366666666" 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG } 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908375" 12:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908375" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG { 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG { 12:46:15 DEBUG "status": "success", 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG { 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG "data": { 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG "data": { 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG "data": { 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG "result": [ 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG "result": [ 12:46:15 DEBUG { 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:46:15 DEBUG "result": [ 12:46:15 DEBUG { 12:46:15 DEBUG "metric": { 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG { 12:46:15 DEBUG "metric": { 12:46:15 DEBUG "instance": "10.142.0.11:9100" 12:46:15 DEBUG { 12:46:15 DEBUG "metric": { 12:46:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 INFO [http_cmd]: http status code OK 12:46:15 DEBUG }, 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG "instance": "10.142.0.11:9100" 12:46:15 DEBUG }, 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG --- status code --- 12:46:15 DEBUG "value": [ 12:46:15 DEBUG "data": { 12:46:15 DEBUG }, 12:46:15 DEBUG "value": [ 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 12:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:46:15 DEBUG 12:46:15 DEBUG "result": [ 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG --- http response --- 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG 12:46:15 DEBUG { 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:46:15 DEBUG { 12:46:15 DEBUG [...] 12:46:15 DEBUG "metric": { 12:46:15 DEBUG 12:46:15 DEBUG [...] 12:46:15 DEBUG ----- output ----- 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG 12:46:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:46:15 DEBUG [...] 12:46:15 DEBUG 12:46:15 DEBUG { 12:46:15 DEBUG "data": { 12:46:15 DEBUG 12:46:15 DEBUG }, 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG "status": "success", 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG "data": { 12:46:15 DEBUG "result": [ 12:46:15 DEBUG "instance": "10.142.0.51:9100" 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:46:15 DEBUG "resultType": "vector", 12:46:15 DEBUG { 12:46:15 DEBUG }, 12:46:15 DEBUG 12:46:15 DEBUG "instance": "10.142.0.51:9100" 12:46:15 DEBUG }, 12:46:15 DEBUG "result": [ 12:46:15 DEBUG "metric": { 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 12:46:15 DEBUG }, 12:46:15 DEBUG "value": [ 12:46:15 DEBUG { 12:46:15 DEBUG "instance": "10.142.0.11:9100" 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG [...] 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG "metric": { 12:46:15 DEBUG }, 12:46:15 DEBUG "1141218.266666667" 12:46:15 DEBUG 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG "0.10827713639180014" 12:46:15 DEBUG "pod": "ds-idrepo-0" 12:46:15 DEBUG "value": [ 12:46:15 DEBUG ] 12:46:15 DEBUG 12:46:15 DEBUG "19387.733333333334" 12:46:15 DEBUG ] 12:46:15 DEBUG }, 12:46:15 DEBUG 12:46:15 DEBUG } 12:46:15 DEBUG 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 12:46:15 DEBUG ] 12:46:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:46:15 DEBUG } 12:46:15 DEBUG ] 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG } 12:46:15 DEBUG }, 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG 12:46:15 DEBUG [...] 12:46:15 DEBUG } 12:46:15 DEBUG "value": [ 12:46:15 DEBUG } 12:46:15 DEBUG } 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG } 12:46:15 DEBUG [...] 12:46:15 DEBUG 12:46:15 DEBUG "14995865600" 12:46:15 DEBUG 12:46:15 DEBUG 12:46:15 DEBUG ] 12:46:15 DEBUG 12:46:15 DEBUG "instance": "10.142.0.51:9100" 12:46:15 DEBUG }, 12:46:15 DEBUG "value": [ 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG 12:46:15 DEBUG } 12:46:15 DEBUG "0" 12:46:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:46:15 DEBUG ] 12:46:15 DEBUG ] 12:46:15 DEBUG }, 12:46:15 DEBUG } 12:46:15 DEBUG } 12:46:15 DEBUG "value": [ 12:46:15 DEBUG } 12:46:15 DEBUG ] 12:46:15 DEBUG 1735908375, 12:46:15 DEBUG } 12:46:15 DEBUG "758779904" 12:46:15 DEBUG } 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG ] 12:46:15 DEBUG } 12:46:15 DEBUG } 12:46:22 INFO 12:46:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:46:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:46:22 INFO [loop_until]: OK (rc = 0) 12:46:22 DEBUG --- stdout --- 12:46:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 175m 1% 3904Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 140m 0% 19056Mi 67% 12:46:22 DEBUG --- stderr --- 12:46:22 DEBUG 12:46:26 INFO 12:46:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:46:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:46:26 INFO [loop_until]: OK (rc = 0) 12:46:26 DEBUG --- stdout --- 12:46:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 56m 13478Mi rcs-645c86fdd6-j4rsk 62m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:46:26 DEBUG --- stderr --- 12:46:26 DEBUG 12:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908434" 12:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908434" 12:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908434" 12:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908434" 12:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908434" 12:47:14 INFO [http_cmd]: http status code OK 12:47:14 DEBUG --- status code --- 12:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:14 DEBUG --- http response --- 12:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:14 ERROR [http_cmd]: ERROR 12:47:14 INFO [http_cmd]: http status code OK 12:47:14 ERROR --- status code --- 12:47:14 INFO [http_cmd]: http status code OK 12:47:14 INFO [http_cmd]: http status code OK 12:47:14 DEBUG --- status code --- 12:47:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:47:14 DEBUG --- status code --- 12:47:14 DEBUG --- status code --- 12:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:14 DEBUG --- http response --- 12:47:14 ERROR --- http response --- 12:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:14 DEBUG --- http response --- 12:47:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:47:14 DEBUG --- http response --- 12:47:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908434" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG { 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG "data": { 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG "result": [ 12:47:15 DEBUG { 12:47:15 DEBUG "metric": { 12:47:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:47:15 DEBUG }, 12:47:15 DEBUG "value": [ 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG [...] 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:47:15 DEBUG }, 12:47:15 DEBUG "value": [ 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG "0" 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG { 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG "data": { 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG "result": [ 12:47:15 DEBUG { 12:47:15 DEBUG { 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG "metric": { 12:47:15 DEBUG "data": { 12:47:15 DEBUG "instance": "10.142.0.11:9100" 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG }, 12:47:15 DEBUG "result": [ 12:47:15 DEBUG "value": [ 12:47:15 DEBUG { 12:47:15 DEBUG 12:47:15 DEBUG "metric": { 12:47:15 DEBUG 12:47:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:47:15 DEBUG 12:47:15 DEBUG }, 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG [...] 12:47:15 DEBUG "value": [ 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 DEBUG "instance": "10.142.0.51:9100" 12:47:15 DEBUG [...] 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG }, 12:47:15 DEBUG 12:47:15 DEBUG { 12:47:15 DEBUG "value": [ 12:47:15 DEBUG 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG 12:47:15 DEBUG "data": { 12:47:15 DEBUG "0.0012000000000019402" 12:47:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG ] 12:47:15 DEBUG }, 12:47:15 DEBUG "result": [ 12:47:15 DEBUG } 12:47:15 DEBUG "value": [ 12:47:15 DEBUG { 12:47:15 DEBUG ] 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG "metric": { 12:47:15 DEBUG } 12:47:15 DEBUG "0.0009609881474912584" 12:47:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:47:15 DEBUG } 12:47:15 DEBUG ] 12:47:15 DEBUG }, 12:47:15 DEBUG } 12:47:15 DEBUG "value": [ 12:47:15 DEBUG ] 12:47:15 DEBUG 12:47:15 DEBUG } 12:47:15 DEBUG 12:47:15 DEBUG } 12:47:15 DEBUG 12:47:15 DEBUG [...] 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:47:15 DEBUG }, 12:47:15 DEBUG "value": [ 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG "457.62945742218164" 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG { 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG "data": { 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG "result": [ 12:47:15 DEBUG { 12:47:15 DEBUG "metric": { 12:47:15 DEBUG "instance": "10.142.0.11:9100" 12:47:15 DEBUG }, 12:47:15 DEBUG "value": [ 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG [...] 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG "instance": "10.142.0.51:9100" 12:47:15 DEBUG }, 12:47:15 DEBUG "value": [ 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG "200210.5666666667" 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG } 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908435" 12:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908435" 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- http response --- 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 DEBUG --- status code --- 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG { 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG "data": { 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG { 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG "result": [ 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG { 12:47:15 DEBUG { 12:47:15 DEBUG "data": { 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG "metric": { 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG "data": { 12:47:15 DEBUG "instance": "10.142.0.11:9100" 12:47:15 DEBUG "result": [ 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG }, 12:47:15 DEBUG { 12:47:15 DEBUG --- http response --- 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG "result": [ 12:47:15 DEBUG "value": [ 12:47:15 DEBUG "metric": { 12:47:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG { 12:47:15 DEBUG 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 DEBUG }, 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG "metric": { 12:47:15 DEBUG 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG "value": [ 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG "instance": "10.142.0.11:9100" 12:47:15 DEBUG 12:47:15 DEBUG { 12:47:15 DEBUG 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:47:15 DEBUG }, 12:47:15 INFO [http_cmd]: http status code OK 12:47:15 DEBUG [...] 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG "value": [ 12:47:15 DEBUG --- status code --- 12:47:15 DEBUG 12:47:15 DEBUG "data": { 12:47:15 DEBUG 12:47:15 DEBUG { 12:47:15 DEBUG 12:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:15 DEBUG 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG [...] 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG 12:47:15 DEBUG --- http response --- 12:47:15 DEBUG 12:47:15 DEBUG "result": [ 12:47:15 DEBUG 12:47:15 DEBUG "data": { 12:47:15 DEBUG 12:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:47:15 DEBUG "instance": "10.142.0.51:9100" 12:47:15 DEBUG { 12:47:15 DEBUG 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG [...] 12:47:15 DEBUG ----- output ----- 12:47:15 DEBUG }, 12:47:15 DEBUG "metric": { 12:47:15 DEBUG 12:47:15 DEBUG "result": [ 12:47:15 DEBUG 12:47:15 DEBUG { 12:47:15 DEBUG "value": [ 12:47:15 DEBUG "instance": "10.142.0.11:9100" 12:47:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:47:15 DEBUG }, 12:47:15 DEBUG "value": [ 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG "0.1281391547964443" 12:47:15 DEBUG }, 12:47:15 DEBUG { 12:47:15 DEBUG 12:47:15 DEBUG "status": "success", 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG ] 12:47:15 DEBUG "value": [ 12:47:15 DEBUG "metric": { 12:47:15 DEBUG 12:47:15 DEBUG "data": { 12:47:15 DEBUG "216978.23333333334" 12:47:15 DEBUG } 12:47:15 DEBUG 12:47:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:47:15 DEBUG "instance": "10.142.0.51:9100" 12:47:15 DEBUG "resultType": "vector", 12:47:15 DEBUG ] 12:47:15 DEBUG ] 12:47:15 DEBUG 12:47:15 DEBUG }, 12:47:15 DEBUG }, 12:47:15 DEBUG "result": [ 12:47:15 DEBUG } 12:47:15 DEBUG } 12:47:15 DEBUG 12:47:15 DEBUG "value": [ 12:47:15 DEBUG "value": [ 12:47:15 DEBUG { 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG [...] 12:47:15 DEBUG 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG "metric": { 12:47:15 DEBUG } 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG "28262.399999999998" 12:47:15 DEBUG "pod": "ds-idrepo-0" 12:47:15 DEBUG } 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG 12:47:15 DEBUG [...] 12:47:15 DEBUG }, 12:47:15 DEBUG ] 12:47:15 DEBUG "instance": "10.142.0.51:9100" 12:47:15 DEBUG 12:47:15 DEBUG "value": [ 12:47:15 DEBUG } 12:47:15 DEBUG }, 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG } 12:47:15 DEBUG "value": [ 12:47:15 DEBUG 12:47:15 DEBUG 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:47:15 DEBUG 12:47:15 DEBUG "0.001333333333333068" 12:47:15 DEBUG }, 12:47:15 DEBUG [...] 12:47:15 DEBUG ] 12:47:15 DEBUG "value": [ 12:47:15 DEBUG 12:47:15 DEBUG } 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG 12:47:15 DEBUG ] 12:47:15 DEBUG "14989066240" 12:47:15 DEBUG 12:47:15 DEBUG } 12:47:15 DEBUG ] 12:47:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:47:15 DEBUG } 12:47:15 DEBUG } 12:47:15 DEBUG }, 12:47:15 DEBUG ] 12:47:15 DEBUG "value": [ 12:47:15 DEBUG } 12:47:15 DEBUG 1735908435, 12:47:15 DEBUG } 12:47:15 DEBUG "758771712" 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG ] 12:47:15 DEBUG } 12:47:15 DEBUG } 12:47:22 INFO 12:47:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:47:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:47:22 INFO [loop_until]: OK (rc = 0) 12:47:22 DEBUG --- stdout --- 12:47:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 139m 0% 3905Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 130m 0% 19050Mi 66% 12:47:22 DEBUG --- stderr --- 12:47:22 DEBUG 12:47:26 INFO 12:47:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:47:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:47:26 INFO [loop_until]: OK (rc = 0) 12:47:26 DEBUG --- stdout --- 12:47:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 6m 13479Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:47:26 DEBUG --- stderr --- 12:47:26 DEBUG 12:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908494" 12:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908494" 12:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908494" 12:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908494" 12:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908494" 12:48:14 INFO [http_cmd]: http status code OK 12:48:14 DEBUG --- status code --- 12:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:14 DEBUG --- http response --- 12:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:14 ERROR [http_cmd]: ERROR 12:48:14 ERROR --- status code --- 12:48:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:48:14 ERROR --- http response --- 12:48:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:48:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:48:14 INFO [http_cmd]: http status code OK 12:48:14 DEBUG --- status code --- 12:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:14 DEBUG --- http response --- 12:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:14 INFO [http_cmd]: http status code OK 12:48:14 INFO [http_cmd]: http status code OK 12:48:14 DEBUG --- status code --- 12:48:14 DEBUG --- status code --- 12:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:14 DEBUG --- http response --- 12:48:14 DEBUG --- http response --- 12:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 DEBUG --- status code --- 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735908495, "0" ] } ] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:48:15 DEBUG ----- output ----- 12:48:15 DEBUG { 12:48:15 DEBUG "status": "success", 12:48:15 DEBUG "data": { 12:48:15 DEBUG "resultType": "vector", 12:48:15 DEBUG "result": [ 12:48:15 DEBUG { 12:48:15 DEBUG "metric": { 12:48:15 DEBUG "instance": "10.142.0.11:9100" 12:48:15 DEBUG }, 12:48:15 DEBUG "value": [ 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG [...] 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG "instance": "10.142.0.51:9100" 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG }, 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG "value": [ 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735908495, "0.0042007241451568235" ] } ] } } 12:48:15 DEBUG 1735908495, 12:48:15 DEBUG "0.0004666666666745793" 12:48:15 DEBUG ] 12:48:15 DEBUG } 12:48:15 DEBUG ] 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735908495, "873.5454029069282" ] } ] } } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:48:15 DEBUG ----- output ----- 12:48:15 DEBUG { 12:48:15 DEBUG "status": "success", 12:48:15 DEBUG "data": { 12:48:15 DEBUG "resultType": "vector", 12:48:15 DEBUG "result": [ 12:48:15 DEBUG { 12:48:15 DEBUG "metric": { 12:48:15 DEBUG "instance": "10.142.0.11:9100" 12:48:15 DEBUG }, 12:48:15 DEBUG "value": [ 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG [...] 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG "instance": "10.142.0.51:9100" 12:48:15 DEBUG }, 12:48:15 DEBUG "value": [ 12:48:15 DEBUG 1735908495, 12:48:15 DEBUG "77863.5" 12:48:15 DEBUG ] 12:48:15 DEBUG } 12:48:15 DEBUG ] 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908495" 12:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908495" 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:48:15 DEBUG ----- output ----- 12:48:15 DEBUG { 12:48:15 DEBUG "status": "success", 12:48:15 DEBUG "data": { 12:48:15 DEBUG "resultType": "vector", 12:48:15 DEBUG "result": [ 12:48:15 DEBUG { 12:48:15 DEBUG "metric": { 12:48:15 DEBUG "instance": "10.142.0.11:9100" 12:48:15 DEBUG }, 12:48:15 DEBUG "value": [ 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG [...] 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG "instance": "10.142.0.51:9100" 12:48:15 DEBUG }, 12:48:15 DEBUG "value": [ 12:48:15 DEBUG 1735908495, 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG "95180.30000000002" 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG ] 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG } 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:48:15 DEBUG ] 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG ----- output ----- 12:48:15 INFO [http_cmd]: http status code OK 12:48:15 DEBUG } 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG ----- output ----- 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { 12:48:15 DEBUG --- status code --- 12:48:15 DEBUG } 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG { 12:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:48:15 DEBUG "status": "success", 12:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:15 DEBUG "status": "success", 12:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:48:15 DEBUG ----- output ----- 12:48:15 DEBUG "data": { 12:48:15 DEBUG --- http response --- 12:48:15 DEBUG "data": { 12:48:15 DEBUG ----- output ----- 12:48:15 DEBUG { 12:48:15 DEBUG "resultType": "vector", 12:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:48:15 DEBUG "resultType": "vector", 12:48:15 DEBUG { 12:48:15 DEBUG "status": "success", 12:48:15 DEBUG "result": [ 12:48:15 DEBUG ----- output ----- 12:48:15 DEBUG "result": [ 12:48:15 DEBUG "status": "success", 12:48:15 DEBUG "data": { 12:48:15 DEBUG { 12:48:15 DEBUG { 12:48:15 DEBUG { 12:48:15 DEBUG "data": { 12:48:15 DEBUG "resultType": "vector", 12:48:15 DEBUG "metric": { 12:48:15 DEBUG "status": "success", 12:48:15 DEBUG "metric": { 12:48:15 DEBUG "resultType": "vector", 12:48:15 DEBUG "result": [ 12:48:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:48:15 DEBUG "data": { 12:48:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:48:15 DEBUG "result": [ 12:48:15 DEBUG { 12:48:15 DEBUG }, 12:48:15 DEBUG "resultType": "vector", 12:48:15 DEBUG }, 12:48:15 DEBUG { 12:48:15 DEBUG "metric": { 12:48:15 DEBUG "value": [ 12:48:15 DEBUG "result": [ 12:48:15 DEBUG "value": [ 12:48:15 DEBUG "metric": { 12:48:15 DEBUG "instance": "10.142.0.11:9100" 12:48:15 DEBUG 12:48:15 DEBUG { 12:48:15 DEBUG 12:48:15 DEBUG "instance": "10.142.0.11:9100" 12:48:15 DEBUG }, 12:48:15 DEBUG 12:48:15 DEBUG "metric": { 12:48:15 DEBUG 12:48:15 DEBUG }, 12:48:15 DEBUG "value": [ 12:48:15 DEBUG 12:48:15 DEBUG "pod": "ds-idrepo-0" 12:48:15 DEBUG 12:48:15 DEBUG "value": [ 12:48:15 DEBUG 12:48:15 DEBUG [...] 12:48:15 DEBUG }, 12:48:15 DEBUG [...] 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG "value": [ 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG [...] 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG [...] 12:48:15 DEBUG 12:48:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:48:15 DEBUG 12:48:15 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG }, 12:48:15 DEBUG [...] 12:48:15 DEBUG 12:48:15 DEBUG }, 12:48:15 DEBUG 12:48:15 DEBUG "value": [ 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG "value": [ 12:48:15 DEBUG "instance": "10.142.0.51:9100" 12:48:15 DEBUG 1735908495, 12:48:15 DEBUG 12:48:15 DEBUG 12:48:15 DEBUG 1735908495, 12:48:15 DEBUG }, 12:48:15 DEBUG "0.11764619133511828" 12:48:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:48:15 DEBUG "instance": "10.142.0.51:9100" 12:48:15 DEBUG "14986141696" 12:48:15 DEBUG "value": [ 12:48:15 DEBUG ] 12:48:15 DEBUG }, 12:48:15 DEBUG }, 12:48:15 DEBUG ] 12:48:15 DEBUG 1735908495, 12:48:15 DEBUG } 12:48:15 DEBUG "value": [ 12:48:15 DEBUG "value": [ 12:48:15 DEBUG } 12:48:15 DEBUG "0.00033333333333350387" 12:48:15 DEBUG ] 12:48:15 DEBUG 1735908495, 12:48:15 DEBUG 1735908495, 12:48:15 DEBUG ] 12:48:15 DEBUG ] 12:48:15 DEBUG } 12:48:15 DEBUG "758779904" 12:48:15 DEBUG "26350.93333333333" 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:15 DEBUG ] 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:15 DEBUG ] 12:48:15 DEBUG ] 12:48:15 DEBUG ] 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:15 DEBUG ] 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:15 DEBUG } 12:48:22 INFO 12:48:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:48:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:48:22 INFO [loop_until]: OK (rc = 0) 12:48:22 DEBUG --- stdout --- 12:48:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3870Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 77m 0% 19051Mi 66% 12:48:22 DEBUG --- stderr --- 12:48:22 DEBUG 12:48:26 INFO 12:48:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:48:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:48:26 INFO [loop_until]: OK (rc = 0) 12:48:26 DEBUG --- stdout --- 12:48:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13478Mi rcs-645c86fdd6-j4rsk 19m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:48:26 DEBUG --- stderr --- 12:48:26 DEBUG 12:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908554" 12:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908554" 12:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908554" 12:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908554" 12:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908554" 12:49:14 INFO [http_cmd]: http status code OK 12:49:14 DEBUG --- status code --- 12:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:14 INFO [http_cmd]: http status code OK 12:49:14 INFO [http_cmd]: http status code OK 12:49:14 DEBUG --- http response --- 12:49:14 ERROR [http_cmd]: ERROR 12:49:14 DEBUG --- status code --- 12:49:14 INFO [http_cmd]: http status code OK 12:49:14 DEBUG --- status code --- 12:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:14 ERROR --- status code --- 12:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:14 DEBUG --- status code --- 12:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:49:14 ERROR --- http response --- 12:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:14 DEBUG --- http response --- 12:49:14 DEBUG --- http response --- 12:49:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:49:14 DEBUG --- http response --- 12:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG { 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG "data": { 12:49:15 DEBUG "resultType": "vector", 12:49:15 DEBUG "result": [ 12:49:15 DEBUG { 12:49:15 DEBUG "metric": { 12:49:15 DEBUG "pod": "ds-cts-0" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG [...] 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 1735908555, 12:49:15 DEBUG "0" 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG { 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG "data": { 12:49:15 DEBUG "resultType": "vector", 12:49:15 DEBUG "result": [ 12:49:15 DEBUG { 12:49:15 DEBUG "metric": { 12:49:15 DEBUG "pod": "ds-cts-0" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG [...] 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 1735908555, 12:49:15 DEBUG "0.0008450121468767891" 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG { 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG "data": { 12:49:15 DEBUG "resultType": "vector", 12:49:15 DEBUG "result": [ 12:49:15 DEBUG { 12:49:15 DEBUG "metric": { 12:49:15 DEBUG "pod": "ds-cts-0" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG [...] 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 1735908555, 12:49:15 DEBUG "673.257294037209" 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG { 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG "data": { 12:49:15 DEBUG "resultType": "vector", 12:49:15 DEBUG "result": [ 12:49:15 DEBUG { 12:49:15 DEBUG "metric": { 12:49:15 DEBUG "instance": "10.142.0.11:9100" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG [...] 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG "instance": "10.142.0.51:9100" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 1735908555, 12:49:15 DEBUG "0.00040000000000190995" 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG { 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG "data": { 12:49:15 DEBUG "resultType": "vector", 12:49:15 DEBUG "result": [ 12:49:15 DEBUG { 12:49:15 DEBUG "metric": { 12:49:15 DEBUG "instance": "10.142.0.11:9100" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG [...] 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG "instance": "10.142.0.51:9100" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 1735908555, 12:49:15 DEBUG "248679.96666666667" 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG } 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908555" 12:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908555" 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG { 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG "data": { 12:49:15 DEBUG "resultType": "vector", 12:49:15 DEBUG "result": [ 12:49:15 DEBUG { 12:49:15 DEBUG "metric": { 12:49:15 DEBUG "instance": "10.142.0.11:9100" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG [...] 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG 12:49:15 DEBUG "instance": "10.142.0.51:9100" 12:49:15 DEBUG }, 12:49:15 DEBUG "value": [ 12:49:15 DEBUG 1735908555, 12:49:15 DEBUG "265333.96666666673" 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG ] 12:49:15 DEBUG } 12:49:15 DEBUG } 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG { 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:49:15 DEBUG "data": { 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG ----- output ----- 12:49:15 INFO [http_cmd]: http status code OK 12:49:15 DEBUG "resultType": "vector", 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG { 12:49:15 DEBUG --- status code --- 12:49:15 DEBUG "result": [ 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:15 DEBUG { 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG "data": { 12:49:15 DEBUG --- http response --- 12:49:15 DEBUG "metric": { 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:49:15 DEBUG "resultType": "vector", 12:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:49:15 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG "result": [ 12:49:15 DEBUG ----- output ----- 12:49:15 DEBUG }, 12:49:15 DEBUG { 12:49:15 DEBUG { 12:49:15 DEBUG { 12:49:15 DEBUG { 12:49:15 DEBUG "value": [ 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG "metric": { 12:49:15 DEBUG "status": "success", 12:49:15 DEBUG 12:49:15 DEBUG "data": { 12:49:15 DEBUG "data": { 12:49:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:49:16 DEBUG "data": { 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG "resultType": "vector", 12:49:16 DEBUG }, 12:49:16 DEBUG "resultType": "vector", 12:49:16 DEBUG "resultType": "vector", 12:49:16 DEBUG [...] 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG "value": [ 12:49:16 DEBUG "result": [ 12:49:16 DEBUG "result": [ 12:49:16 DEBUG 12:49:16 DEBUG "result": [ 12:49:16 DEBUG 12:49:16 DEBUG { 12:49:16 DEBUG { 12:49:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:49:16 DEBUG { 12:49:16 DEBUG 12:49:16 DEBUG "metric": { 12:49:16 DEBUG "metric": { 12:49:16 DEBUG }, 12:49:16 DEBUG "metric": { 12:49:16 DEBUG 12:49:16 DEBUG "instance": "10.142.0.11:9100" 12:49:16 DEBUG "instance": "10.142.0.11:9100" 12:49:16 DEBUG "value": [ 12:49:16 DEBUG "pod": "ds-idrepo-0" 12:49:16 DEBUG [...] 12:49:16 DEBUG }, 12:49:16 DEBUG }, 12:49:16 DEBUG 1735908555, 12:49:16 DEBUG }, 12:49:16 DEBUG 12:49:16 DEBUG "value": [ 12:49:16 DEBUG "value": [ 12:49:16 DEBUG "14984089600" 12:49:16 DEBUG "value": [ 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG ] 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG } 12:49:16 DEBUG 12:49:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG ] 12:49:16 DEBUG 12:49:16 DEBUG }, 12:49:16 DEBUG [...] 12:49:16 DEBUG [...] 12:49:16 DEBUG } 12:49:16 DEBUG [...] 12:49:16 DEBUG "value": [ 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG } 12:49:16 DEBUG 12:49:16 DEBUG 1735908555, 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG "0.10885708100959587" 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG 12:49:16 DEBUG ] 12:49:16 DEBUG "instance": "10.142.0.51:9100" 12:49:16 DEBUG "instance": "10.142.0.51:9100" 12:49:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:49:16 DEBUG } 12:49:16 DEBUG }, 12:49:16 DEBUG }, 12:49:16 DEBUG }, 12:49:16 DEBUG ] 12:49:16 DEBUG "value": [ 12:49:16 DEBUG "value": [ 12:49:16 DEBUG "value": [ 12:49:16 DEBUG } 12:49:16 DEBUG 1735908555, 12:49:16 DEBUG 1735908555, 12:49:16 DEBUG 1735908555, 12:49:16 DEBUG } 12:49:16 DEBUG "26077.866666666665" 12:49:16 DEBUG "0.0003333333333330302" 12:49:16 DEBUG "758771712" 12:49:16 DEBUG ] 12:49:16 DEBUG ] 12:49:16 DEBUG ] 12:49:16 DEBUG } 12:49:16 DEBUG } 12:49:16 DEBUG } 12:49:16 DEBUG ] 12:49:16 DEBUG ] 12:49:16 DEBUG ] 12:49:16 DEBUG } 12:49:16 DEBUG } 12:49:16 DEBUG } 12:49:16 DEBUG } 12:49:16 DEBUG } 12:49:16 DEBUG } 12:49:22 INFO 12:49:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:49:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:49:22 INFO [loop_until]: OK (rc = 0) 12:49:22 DEBUG --- stdout --- 12:49:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3854Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 415m 1% 19053Mi 66% 12:49:22 DEBUG --- stderr --- 12:49:22 DEBUG 12:49:26 INFO 12:49:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:49:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:49:26 INFO [loop_until]: OK (rc = 0) 12:49:26 DEBUG --- stdout --- 12:49:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 526m 13479Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:49:26 DEBUG --- stderr --- 12:49:26 DEBUG 12:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908614" 12:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908614" 12:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908614" 12:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908614" 12:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908614" 12:50:14 INFO [http_cmd]: http status code OK 12:50:14 DEBUG --- status code --- 12:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:14 DEBUG --- http response --- 12:50:14 ERROR [http_cmd]: ERROR 12:50:14 ERROR --- status code --- 12:50:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:50:14 ERROR --- http response --- 12:50:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:50:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:50:14 INFO [http_cmd]: http status code OK 12:50:14 DEBUG --- status code --- 12:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:14 DEBUG --- http response --- 12:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:14 INFO [http_cmd]: http status code OK 12:50:14 DEBUG --- status code --- 12:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:14 DEBUG --- http response --- 12:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:14 INFO [http_cmd]: http status code OK 12:50:14 DEBUG --- status code --- 12:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:14 DEBUG --- http response --- 12:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:15 DEBUG ----- output ----- 12:50:15 DEBUG { 12:50:15 DEBUG "status": "success", 12:50:15 DEBUG "data": { 12:50:15 DEBUG "resultType": "vector", 12:50:15 DEBUG "result": [ 12:50:15 DEBUG { 12:50:15 DEBUG "metric": { 12:50:15 DEBUG "pod": "ds-cts-0" 12:50:15 DEBUG }, 12:50:15 DEBUG "value": [ 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG [...] 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:50:15 DEBUG }, 12:50:15 DEBUG "value": [ 12:50:15 DEBUG 1735908615, 12:50:15 DEBUG "0" 12:50:15 DEBUG ] 12:50:15 DEBUG } 12:50:15 DEBUG ] 12:50:15 DEBUG } 12:50:15 DEBUG } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908615" 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:50:15 DEBUG ----- output ----- 12:50:15 DEBUG { 12:50:15 DEBUG "status": "success", 12:50:15 DEBUG "data": { 12:50:15 DEBUG "resultType": "vector", 12:50:15 DEBUG "result": [ 12:50:15 DEBUG { 12:50:15 DEBUG "metric": { 12:50:15 DEBUG "pod": "ds-cts-0" 12:50:15 DEBUG }, 12:50:15 DEBUG "value": [ 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG [...] 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG "pod": "ds-idrepo-0" 12:50:15 DEBUG }, 12:50:15 DEBUG "value": [ 12:50:15 DEBUG 1735908615, 12:50:15 DEBUG "0" 12:50:15 DEBUG ] 12:50:15 DEBUG } 12:50:15 DEBUG ] 12:50:15 DEBUG } 12:50:15 DEBUG } 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG ----- output ----- 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG { 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG "status": "success", 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG "data": { 12:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:50:15 DEBUG "resultType": "vector", 12:50:15 DEBUG ----- output ----- 12:50:15 DEBUG "result": [ 12:50:15 DEBUG { 12:50:15 DEBUG { 12:50:15 DEBUG "status": "success", 12:50:15 DEBUG "metric": { 12:50:15 DEBUG "data": { 12:50:15 DEBUG "pod": "ds-cts-0" 12:50:15 DEBUG "resultType": "vector", 12:50:15 DEBUG }, 12:50:15 DEBUG "result": [ 12:50:15 DEBUG "value": [ 12:50:15 DEBUG { 12:50:15 DEBUG 12:50:15 DEBUG "metric": { 12:50:15 DEBUG 12:50:15 DEBUG "instance": "10.142.0.11:9100" 12:50:15 DEBUG 12:50:15 DEBUG }, 12:50:15 DEBUG [...] 12:50:15 DEBUG "value": [ 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:50:15 DEBUG [...] 12:50:15 DEBUG }, 12:50:15 DEBUG 12:50:15 DEBUG "value": [ 12:50:15 DEBUG 12:50:15 DEBUG 1735908615, 12:50:15 DEBUG 12:50:15 DEBUG "363.4427684117125" 12:50:15 DEBUG "instance": "10.142.0.51:9100" 12:50:15 DEBUG ] 12:50:15 DEBUG }, 12:50:15 DEBUG } 12:50:15 DEBUG "value": [ 12:50:15 DEBUG ] 12:50:15 DEBUG 1735908615, 12:50:15 DEBUG } 12:50:15 DEBUG "0.00033333333333681974" 12:50:15 DEBUG } 12:50:15 DEBUG ] 12:50:15 DEBUG } 12:50:15 DEBUG ] 12:50:15 DEBUG } 12:50:15 DEBUG } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908615" 12:50:15 INFO [http_cmd]: http status code OK 12:50:15 DEBUG --- status code --- 12:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:15 DEBUG --- http response --- 12:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:50:15 DEBUG ----- output ----- 12:50:15 DEBUG { 12:50:15 DEBUG "status": "success", 12:50:15 DEBUG "data": { 12:50:15 DEBUG "resultType": "vector", 12:50:15 DEBUG "result": [ 12:50:15 DEBUG { 12:50:15 DEBUG "metric": { 12:50:15 DEBUG "instance": "10.142.0.11:9100" 12:50:15 DEBUG }, 12:50:15 DEBUG "value": [ 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG [...] 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG 12:50:15 DEBUG "instance": "10.142.0.51:9100" 12:50:15 DEBUG }, 12:50:15 DEBUG "value": [ 12:50:15 DEBUG 1735908615, 12:50:15 DEBUG "1235284.7666666661" 12:50:15 DEBUG ] 12:50:15 DEBUG } 12:50:15 DEBUG ] 12:50:15 DEBUG } 12:50:15 DEBUG } 12:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908615" 12:50:16 INFO [http_cmd]: http status code OK 12:50:16 DEBUG --- status code --- 12:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:16 DEBUG --- http response --- 12:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:50:16 DEBUG ----- output ----- 12:50:16 DEBUG { 12:50:16 DEBUG "status": "success", 12:50:16 DEBUG "data": { 12:50:16 DEBUG "resultType": "vector", 12:50:16 DEBUG "result": [ 12:50:16 DEBUG { 12:50:16 DEBUG "metric": { 12:50:16 DEBUG "instance": "10.142.0.11:9100" 12:50:16 DEBUG }, 12:50:16 DEBUG "value": [ 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG [...] 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG "instance": "10.142.0.51:9100" 12:50:16 DEBUG }, 12:50:16 DEBUG "value": [ 12:50:16 DEBUG 1735908615, 12:50:16 DEBUG "1252746.066666667" 12:50:16 DEBUG ] 12:50:16 DEBUG } 12:50:16 DEBUG ] 12:50:16 DEBUG } 12:50:16 DEBUG } 12:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908616" 12:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908616" 12:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908616" 12:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908616" 12:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908616" 12:50:16 INFO [http_cmd]: http status code OK 12:50:16 DEBUG --- status code --- 12:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:16 DEBUG --- http response --- 12:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:50:16 DEBUG ----- output ----- 12:50:16 DEBUG { 12:50:16 DEBUG "status": "success", 12:50:16 DEBUG "data": { 12:50:16 DEBUG "resultType": "vector", 12:50:16 DEBUG "result": [ 12:50:16 DEBUG { 12:50:16 DEBUG "metric": { 12:50:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:50:16 DEBUG }, 12:50:16 DEBUG "value": [ 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG [...] 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:50:16 DEBUG }, 12:50:16 DEBUG "value": [ 12:50:16 DEBUG 1735908616, 12:50:16 DEBUG "0.1275037775136685" 12:50:16 DEBUG ] 12:50:16 DEBUG } 12:50:16 DEBUG ] 12:50:16 DEBUG } 12:50:16 DEBUG } 12:50:16 INFO [http_cmd]: http status code OK 12:50:16 INFO [http_cmd]: http status code OK 12:50:16 DEBUG --- status code --- 12:50:16 INFO [http_cmd]: http status code OK 12:50:16 DEBUG --- status code --- 12:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:16 DEBUG --- status code --- 12:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:16 DEBUG --- http response --- 12:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:16 DEBUG --- http response --- 12:50:16 DEBUG --- http response --- 12:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:50:16 DEBUG ----- output ----- 12:50:16 INFO [http_cmd]: http status code OK 12:50:16 DEBUG ----- output ----- 12:50:16 DEBUG ----- output ----- 12:50:16 DEBUG { 12:50:16 DEBUG --- status code --- 12:50:16 DEBUG { 12:50:16 DEBUG { 12:50:16 DEBUG "status": "success", 12:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:16 DEBUG "status": "success", 12:50:16 DEBUG "status": "success", 12:50:16 DEBUG "data": { 12:50:16 DEBUG "resultType": "vector", 12:50:16 DEBUG "data": { 12:50:16 DEBUG "data": { 12:50:16 DEBUG --- http response --- 12:50:16 DEBUG "result": [ 12:50:16 DEBUG "resultType": "vector", 12:50:16 DEBUG "resultType": "vector", 12:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:50:16 DEBUG { 12:50:16 DEBUG "result": [ 12:50:16 DEBUG "result": [ 12:50:16 DEBUG ----- output ----- 12:50:16 DEBUG "metric": { 12:50:16 DEBUG { 12:50:16 DEBUG { 12:50:16 DEBUG { 12:50:16 DEBUG "instance": "10.142.0.11:9100" 12:50:16 DEBUG "metric": { 12:50:16 DEBUG "metric": { 12:50:16 DEBUG "status": "success", 12:50:16 DEBUG }, 12:50:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:50:16 DEBUG "instance": "10.142.0.11:9100" 12:50:16 DEBUG "data": { 12:50:16 DEBUG "value": [ 12:50:16 DEBUG }, 12:50:16 DEBUG }, 12:50:16 DEBUG "resultType": "vector", 12:50:16 DEBUG 12:50:16 DEBUG "value": [ 12:50:16 DEBUG "value": [ 12:50:16 DEBUG "result": [ 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG { 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG "metric": { 12:50:16 DEBUG [...] 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG "pod": "ds-idrepo-0" 12:50:16 DEBUG 12:50:16 DEBUG [...] 12:50:16 DEBUG [...] 12:50:16 DEBUG }, 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG "value": [ 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG "instance": "10.142.0.51:9100" 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG 12:50:16 DEBUG }, 12:50:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:50:16 DEBUG "instance": "10.142.0.51:9100" 12:50:16 DEBUG 12:50:16 DEBUG "value": [ 12:50:16 DEBUG }, 12:50:16 DEBUG }, 12:50:16 DEBUG [...] 12:50:16 DEBUG 1735908616, 12:50:16 DEBUG "value": [ 12:50:16 DEBUG "value": [ 12:50:16 DEBUG 12:50:16 DEBUG "0" 12:50:16 DEBUG 1735908616, 12:50:16 DEBUG 1735908616, 12:50:16 DEBUG 12:50:16 DEBUG ] 12:50:16 DEBUG "14989438976" 12:50:16 DEBUG "13926.400000000001" 12:50:16 DEBUG ] 12:50:16 DEBUG 12:50:16 DEBUG ] 12:50:16 DEBUG } 12:50:16 DEBUG } 12:50:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:50:16 DEBUG } 12:50:16 DEBUG ] 12:50:16 DEBUG ] 12:50:16 DEBUG }, 12:50:16 DEBUG ] 12:50:16 DEBUG } 12:50:16 DEBUG } 12:50:16 DEBUG } 12:50:16 DEBUG } 12:50:16 DEBUG } 12:50:16 DEBUG "value": [ 12:50:16 DEBUG } 12:50:16 DEBUG 1735908616, 12:50:16 DEBUG "758771712" 12:50:16 DEBUG ] 12:50:16 DEBUG } 12:50:16 DEBUG ] 12:50:16 DEBUG } 12:50:16 DEBUG } 12:50:22 INFO 12:50:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:50:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:50:22 INFO [loop_until]: OK (rc = 0) 12:50:22 DEBUG --- stdout --- 12:50:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 158m 0% 3877Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 119m 0% 19057Mi 67% 12:50:22 DEBUG --- stderr --- 12:50:22 DEBUG 12:50:26 INFO 12:50:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:50:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:50:26 INFO [loop_until]: OK (rc = 0) 12:50:26 DEBUG --- stdout --- 12:50:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13479Mi rcs-645c86fdd6-j4rsk 65m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:50:26 DEBUG --- stderr --- 12:50:26 DEBUG 12:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908674" 12:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908674" 12:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908674" 12:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908674" 12:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908674" 12:51:14 INFO [http_cmd]: http status code OK 12:51:14 DEBUG --- status code --- 12:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:14 DEBUG --- http response --- 12:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:14 INFO [http_cmd]: http status code OK 12:51:14 DEBUG --- status code --- 12:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:14 DEBUG --- http response --- 12:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:14 INFO [http_cmd]: http status code OK 12:51:14 DEBUG --- status code --- 12:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:14 DEBUG --- http response --- 12:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:14 INFO [http_cmd]: http status code OK 12:51:14 DEBUG --- status code --- 12:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:14 DEBUG --- http response --- 12:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:14 ERROR [http_cmd]: ERROR 12:51:14 ERROR --- status code --- 12:51:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:51:14 ERROR --- http response --- 12:51:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:51:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG --- http response --- 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:15 DEBUG ----- output ----- 12:51:15 DEBUG { 12:51:15 DEBUG "status": "success", 12:51:15 DEBUG "data": { 12:51:15 DEBUG "resultType": "vector", 12:51:15 DEBUG "result": [ 12:51:15 DEBUG { 12:51:15 DEBUG "metric": { 12:51:15 DEBUG "pod": "ds-idrepo-0" 12:51:15 DEBUG }, 12:51:15 DEBUG "value": [ 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG [...] 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:51:15 DEBUG }, 12:51:15 DEBUG "value": [ 12:51:15 DEBUG 1735908675, 12:51:15 DEBUG "0" 12:51:15 DEBUG ] 12:51:15 DEBUG } 12:51:15 DEBUG ] 12:51:15 DEBUG } 12:51:15 DEBUG } 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908675" 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:15 DEBUG ----- output ----- 12:51:15 DEBUG { 12:51:15 DEBUG "status": "success", 12:51:15 DEBUG "data": { 12:51:15 DEBUG "resultType": "vector", 12:51:15 DEBUG "result": [ 12:51:15 DEBUG { 12:51:15 DEBUG "metric": { 12:51:15 DEBUG "pod": "ds-idrepo-0" 12:51:15 DEBUG }, 12:51:15 DEBUG "value": [ 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG [...] 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:51:15 DEBUG }, 12:51:15 DEBUG "value": [ 12:51:15 DEBUG 1735908675, 12:51:15 DEBUG "0.04868079493925975" 12:51:15 DEBUG ] 12:51:15 DEBUG } 12:51:15 DEBUG ] 12:51:15 DEBUG } 12:51:15 DEBUG } 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:15 DEBUG ----- output ----- 12:51:15 DEBUG { 12:51:15 DEBUG "status": "success", 12:51:15 DEBUG "data": { 12:51:15 DEBUG "resultType": "vector", 12:51:15 DEBUG "result": [ 12:51:15 DEBUG { 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG "metric": { 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG "pod": "ds-idrepo-0" 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG }, 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG "value": [ 12:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:51:15 DEBUG 12:51:15 DEBUG ----- output ----- 12:51:15 DEBUG 12:51:15 DEBUG { 12:51:15 DEBUG 12:51:15 DEBUG "status": "success", 12:51:15 DEBUG [...] 12:51:15 DEBUG "data": { 12:51:15 DEBUG 12:51:15 DEBUG "resultType": "vector", 12:51:15 DEBUG 12:51:15 DEBUG "result": [ 12:51:15 DEBUG 12:51:15 DEBUG { 12:51:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:51:15 DEBUG "metric": { 12:51:15 DEBUG }, 12:51:15 DEBUG "instance": "10.142.0.11:9100" 12:51:15 DEBUG "value": [ 12:51:15 DEBUG }, 12:51:15 DEBUG 1735908675, 12:51:15 DEBUG "value": [ 12:51:15 DEBUG "463.906258762415" 12:51:15 DEBUG 12:51:15 DEBUG ] 12:51:15 DEBUG 12:51:15 DEBUG } 12:51:15 DEBUG 12:51:15 DEBUG ] 12:51:15 DEBUG [...] 12:51:15 DEBUG } 12:51:15 DEBUG 12:51:15 DEBUG } 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG "instance": "10.142.0.51:9100" 12:51:15 DEBUG }, 12:51:15 DEBUG "value": [ 12:51:15 DEBUG 1735908675, 12:51:15 DEBUG "0.0004999999999967031" 12:51:15 DEBUG ] 12:51:15 DEBUG } 12:51:15 DEBUG ] 12:51:15 DEBUG } 12:51:15 DEBUG } 12:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908675" 12:51:15 INFO [http_cmd]: http status code OK 12:51:15 DEBUG --- status code --- 12:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:15 DEBUG --- http response --- 12:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:51:15 DEBUG ----- output ----- 12:51:15 DEBUG { 12:51:15 DEBUG "status": "success", 12:51:15 DEBUG "data": { 12:51:15 DEBUG "resultType": "vector", 12:51:15 DEBUG "result": [ 12:51:15 DEBUG { 12:51:15 DEBUG "metric": { 12:51:15 DEBUG "instance": "10.142.0.11:9100" 12:51:15 DEBUG }, 12:51:15 DEBUG "value": [ 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG [...] 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG 12:51:15 DEBUG "instance": "10.142.0.51:9100" 12:51:15 DEBUG }, 12:51:15 DEBUG "value": [ 12:51:15 DEBUG 1735908675, 12:51:15 DEBUG "1235887.4999999995" 12:51:15 DEBUG ] 12:51:15 DEBUG } 12:51:15 DEBUG ] 12:51:15 DEBUG } 12:51:15 DEBUG } 12:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908676" 12:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908676" 12:51:16 INFO [http_cmd]: http status code OK 12:51:16 DEBUG --- status code --- 12:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:16 DEBUG --- http response --- 12:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:51:16 DEBUG ----- output ----- 12:51:16 DEBUG { 12:51:16 DEBUG "status": "success", 12:51:16 DEBUG "data": { 12:51:16 DEBUG "resultType": "vector", 12:51:16 DEBUG "result": [ 12:51:16 DEBUG { 12:51:16 DEBUG "metric": { 12:51:16 DEBUG "instance": "10.142.0.11:9100" 12:51:16 DEBUG }, 12:51:16 DEBUG "value": [ 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG [...] 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG "instance": "10.142.0.51:9100" 12:51:16 DEBUG }, 12:51:16 DEBUG "value": [ 12:51:16 DEBUG 1735908676, 12:51:16 DEBUG "1252963.2333333334" 12:51:16 DEBUG ] 12:51:16 DEBUG } 12:51:16 DEBUG ] 12:51:16 DEBUG } 12:51:16 DEBUG } 12:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908676" 12:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908676" 12:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908676" 12:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908676" 12:51:16 INFO [http_cmd]: http status code OK 12:51:16 DEBUG --- status code --- 12:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:16 DEBUG --- http response --- 12:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:51:16 DEBUG ----- output ----- 12:51:16 DEBUG { 12:51:16 DEBUG "status": "success", 12:51:16 DEBUG "data": { 12:51:16 DEBUG "resultType": "vector", 12:51:16 DEBUG "result": [ 12:51:16 DEBUG { 12:51:16 DEBUG "metric": { 12:51:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:51:16 DEBUG }, 12:51:16 DEBUG "value": [ 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG [...] 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:51:16 DEBUG }, 12:51:16 DEBUG "value": [ 12:51:16 DEBUG 1735908676, 12:51:16 DEBUG "0.12272651765140136" 12:51:16 DEBUG ] 12:51:16 DEBUG } 12:51:16 DEBUG ] 12:51:16 DEBUG } 12:51:16 DEBUG } 12:51:16 INFO [http_cmd]: http status code OK 12:51:16 DEBUG --- status code --- 12:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:16 DEBUG --- http response --- 12:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:51:16 DEBUG ----- output ----- 12:51:16 DEBUG { 12:51:16 DEBUG "status": "success", 12:51:16 DEBUG "data": { 12:51:16 DEBUG "resultType": "vector", 12:51:16 DEBUG "result": [ 12:51:16 DEBUG { 12:51:16 DEBUG "metric": { 12:51:16 DEBUG "instance": "10.142.0.11:9100" 12:51:16 DEBUG }, 12:51:16 DEBUG "value": [ 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG [...] 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG "instance": "10.142.0.51:9100" 12:51:16 DEBUG }, 12:51:16 DEBUG "value": [ 12:51:16 DEBUG 1735908676, 12:51:16 DEBUG "25668.266666666666" 12:51:16 DEBUG ] 12:51:16 DEBUG } 12:51:16 DEBUG ] 12:51:16 DEBUG } 12:51:16 DEBUG } 12:51:16 INFO [http_cmd]: http status code OK 12:51:16 DEBUG --- status code --- 12:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:16 DEBUG --- http response --- 12:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:51:16 DEBUG ----- output ----- 12:51:16 DEBUG { 12:51:16 DEBUG "status": "success", 12:51:16 INFO [http_cmd]: http status code OK 12:51:16 DEBUG --- status code --- 12:51:16 INFO [http_cmd]: http status code OK 12:51:16 DEBUG "data": { 12:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:16 DEBUG --- status code --- 12:51:16 DEBUG "resultType": "vector", 12:51:16 DEBUG --- http response --- 12:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:16 DEBUG "result": [ 12:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:51:16 DEBUG --- http response --- 12:51:16 DEBUG { 12:51:16 DEBUG ----- output ----- 12:51:16 DEBUG "metric": { 12:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:51:16 DEBUG { 12:51:16 DEBUG "pod": "ds-idrepo-0" 12:51:16 DEBUG ----- output ----- 12:51:16 DEBUG "status": "success", 12:51:16 DEBUG }, 12:51:16 DEBUG { 12:51:16 DEBUG "data": { 12:51:16 DEBUG "value": [ 12:51:16 DEBUG "status": "success", 12:51:16 DEBUG "resultType": "vector", 12:51:16 DEBUG 12:51:16 DEBUG "data": { 12:51:16 DEBUG "result": [ 12:51:16 DEBUG 12:51:16 DEBUG "resultType": "vector", 12:51:16 DEBUG { 12:51:16 DEBUG 12:51:16 DEBUG "result": [ 12:51:16 DEBUG "metric": { 12:51:16 DEBUG [...] 12:51:16 DEBUG { 12:51:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:51:16 DEBUG 12:51:16 DEBUG "metric": { 12:51:16 DEBUG }, 12:51:16 DEBUG 12:51:16 DEBUG "instance": "10.142.0.11:9100" 12:51:16 DEBUG "value": [ 12:51:16 DEBUG 12:51:16 DEBUG }, 12:51:16 DEBUG 12:51:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:51:16 DEBUG "value": [ 12:51:16 DEBUG 12:51:16 DEBUG }, 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG "value": [ 12:51:16 DEBUG 12:51:16 DEBUG [...] 12:51:16 DEBUG 1735908676, 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG "758771712" 12:51:16 DEBUG [...] 12:51:16 DEBUG 12:51:16 DEBUG ] 12:51:16 DEBUG 12:51:16 DEBUG 12:51:16 DEBUG } 12:51:16 DEBUG 12:51:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:51:16 DEBUG ] 12:51:16 DEBUG 12:51:16 DEBUG }, 12:51:16 DEBUG } 12:51:16 DEBUG "instance": "10.142.0.51:9100" 12:51:16 DEBUG "value": [ 12:51:16 DEBUG } 12:51:16 DEBUG }, 12:51:16 DEBUG 1735908676, 12:51:16 DEBUG "value": [ 12:51:16 DEBUG "14990069760" 12:51:16 DEBUG 1735908676, 12:51:16 DEBUG ] 12:51:16 DEBUG "0.0010000000000005115" 12:51:16 DEBUG } 12:51:16 DEBUG ] 12:51:16 DEBUG ] 12:51:16 DEBUG } 12:51:16 DEBUG } 12:51:16 DEBUG ] 12:51:16 DEBUG } 12:51:16 DEBUG } 12:51:16 DEBUG } 12:51:22 INFO 12:51:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:51:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:51:22 INFO [loop_until]: OK (rc = 0) 12:51:22 DEBUG --- stdout --- 12:51:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 170m 1% 3862Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 148m 0% 19058Mi 67% 12:51:22 DEBUG --- stderr --- 12:51:22 DEBUG 12:51:26 INFO 12:51:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:51:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:51:26 INFO [loop_until]: OK (rc = 0) 12:51:26 DEBUG --- stdout --- 12:51:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 79m 13479Mi rcs-645c86fdd6-j4rsk 108m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:51:26 DEBUG --- stderr --- 12:51:26 DEBUG 12:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908734" 12:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908734" 12:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908734" 12:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908734" 12:52:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908734" 12:52:14 INFO [http_cmd]: http status code OK 12:52:14 DEBUG --- status code --- 12:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:14 DEBUG --- http response --- 12:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:14 ERROR [http_cmd]: ERROR 12:52:14 ERROR --- status code --- 12:52:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:52:14 ERROR --- http response --- 12:52:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:52:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:52:14 INFO [http_cmd]: http status code OK 12:52:14 DEBUG --- status code --- 12:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:14 DEBUG --- http response --- 12:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:14 INFO [http_cmd]: http status code OK 12:52:14 DEBUG --- status code --- 12:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:14 DEBUG --- http response --- 12:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:14 INFO [http_cmd]: http status code OK 12:52:14 DEBUG --- status code --- 12:52:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:14 DEBUG --- http response --- 12:52:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735908735, "0" ] } ] } } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908735" 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735908735, "0.00040850766252287023" ] } ] } } 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735908735, "340.52928275525954" ] } ] } } 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:52:15 DEBUG ----- output ----- 12:52:15 DEBUG { 12:52:15 DEBUG "status": "success", 12:52:15 DEBUG "data": { 12:52:15 DEBUG "resultType": "vector", 12:52:15 DEBUG "result": [ 12:52:15 DEBUG { 12:52:15 DEBUG "metric": { 12:52:15 DEBUG "instance": "10.142.0.11:9100" 12:52:15 DEBUG }, 12:52:15 DEBUG "value": [ 12:52:15 DEBUG 12:52:15 DEBUG 12:52:15 DEBUG 12:52:15 DEBUG [...] 12:52:15 DEBUG 12:52:15 DEBUG 12:52:15 DEBUG 12:52:15 DEBUG "instance": "10.142.0.51:9100" 12:52:15 DEBUG }, 12:52:15 DEBUG "value": [ 12:52:15 DEBUG 1735908735, 12:52:15 DEBUG "0.0011000000000005153" 12:52:15 DEBUG ] 12:52:15 DEBUG } 12:52:15 DEBUG ] 12:52:15 DEBUG } 12:52:15 DEBUG } 12:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908735" 12:52:15 INFO [http_cmd]: http status code OK 12:52:15 DEBUG --- status code --- 12:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:15 DEBUG --- http response --- 12:52:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:52:15 DEBUG ----- output ----- 12:52:15 DEBUG { 12:52:15 DEBUG "status": "success", 12:52:15 DEBUG "data": { 12:52:15 DEBUG "resultType": "vector", 12:52:15 DEBUG "result": [ 12:52:15 DEBUG { 12:52:15 DEBUG "metric": { 12:52:15 DEBUG "instance": "10.142.0.11:9100" 12:52:15 DEBUG }, 12:52:15 DEBUG "value": [ 12:52:15 DEBUG 12:52:15 DEBUG 12:52:15 DEBUG 12:52:15 DEBUG [...] 12:52:15 DEBUG 12:52:15 DEBUG 12:52:15 DEBUG 12:52:15 DEBUG "instance": "10.142.0.51:9100" 12:52:15 DEBUG }, 12:52:15 DEBUG "value": [ 12:52:15 DEBUG 1735908735, 12:52:15 DEBUG "2119464.5666666664" 12:52:15 DEBUG ] 12:52:15 DEBUG } 12:52:15 DEBUG ] 12:52:15 DEBUG } 12:52:15 DEBUG } 12:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908736" 12:52:16 INFO [http_cmd]: http status code OK 12:52:16 DEBUG --- status code --- 12:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:16 DEBUG --- http response --- 12:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:52:16 DEBUG ----- output ----- 12:52:16 DEBUG { 12:52:16 DEBUG "status": "success", 12:52:16 DEBUG "data": { 12:52:16 DEBUG "resultType": "vector", 12:52:16 DEBUG "result": [ 12:52:16 DEBUG { 12:52:16 DEBUG "metric": { 12:52:16 DEBUG "instance": "10.142.0.11:9100" 12:52:16 DEBUG }, 12:52:16 DEBUG "value": [ 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG [...] 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG "instance": "10.142.0.51:9100" 12:52:16 DEBUG }, 12:52:16 DEBUG "value": [ 12:52:16 DEBUG 1735908736, 12:52:16 DEBUG "2136646.3000000003" 12:52:16 DEBUG ] 12:52:16 DEBUG } 12:52:16 DEBUG ] 12:52:16 DEBUG } 12:52:16 DEBUG } 12:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908736" 12:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908736" 12:52:16 INFO [http_cmd]: http status code OK 12:52:16 DEBUG --- status code --- 12:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:16 INFO [http_cmd]: http status code OK 12:52:16 DEBUG --- http response --- 12:52:16 DEBUG --- status code --- 12:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:16 DEBUG ----- output ----- 12:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908736" 12:52:16 DEBUG --- http response --- 12:52:16 DEBUG { 12:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908736" 12:52:16 DEBUG "status": "success", 12:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908736" 12:52:16 DEBUG ----- output ----- 12:52:16 DEBUG "data": { 12:52:16 DEBUG "resultType": "vector", 12:52:16 DEBUG "result": [ 12:52:16 DEBUG { 12:52:16 DEBUG { 12:52:16 DEBUG "status": "success", 12:52:16 DEBUG "metric": { 12:52:16 DEBUG "data": { 12:52:16 DEBUG "instance": "10.142.0.11:9100" 12:52:16 DEBUG "resultType": "vector", 12:52:16 DEBUG }, 12:52:16 DEBUG "result": [ 12:52:16 DEBUG "value": [ 12:52:16 DEBUG { 12:52:16 DEBUG 12:52:16 DEBUG "metric": { 12:52:16 DEBUG 12:52:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:52:16 DEBUG 12:52:16 DEBUG }, 12:52:16 DEBUG [...] 12:52:16 DEBUG "value": [ 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG "instance": "10.142.0.51:9100" 12:52:16 DEBUG [...] 12:52:16 DEBUG }, 12:52:16 DEBUG 12:52:16 DEBUG "value": [ 12:52:16 DEBUG 12:52:16 DEBUG 1735908736, 12:52:16 DEBUG 12:52:16 DEBUG "24576" 12:52:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:52:16 DEBUG ] 12:52:16 DEBUG }, 12:52:16 DEBUG } 12:52:16 DEBUG "value": [ 12:52:16 DEBUG ] 12:52:16 DEBUG 1735908736, 12:52:16 DEBUG } 12:52:16 DEBUG "0.04385711256490395" 12:52:16 DEBUG } 12:52:16 DEBUG ] 12:52:16 DEBUG } 12:52:16 DEBUG ] 12:52:16 DEBUG } 12:52:16 DEBUG } 12:52:16 INFO [http_cmd]: http status code OK 12:52:16 INFO [http_cmd]: http status code OK 12:52:16 DEBUG --- status code --- 12:52:16 DEBUG --- status code --- 12:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:16 DEBUG --- http response --- 12:52:16 DEBUG --- http response --- 12:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:52:16 DEBUG ----- output ----- 12:52:16 DEBUG ----- output ----- 12:52:16 DEBUG { 12:52:16 DEBUG { 12:52:16 DEBUG "status": "success", 12:52:16 DEBUG "status": "success", 12:52:16 DEBUG "data": { 12:52:16 DEBUG "data": { 12:52:16 DEBUG "resultType": "vector", 12:52:16 DEBUG "resultType": "vector", 12:52:16 DEBUG "result": [ 12:52:16 DEBUG "result": [ 12:52:16 DEBUG { 12:52:16 DEBUG { 12:52:16 DEBUG "metric": { 12:52:16 DEBUG "metric": { 12:52:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:52:16 DEBUG "pod": "ds-idrepo-0" 12:52:16 DEBUG }, 12:52:16 DEBUG }, 12:52:16 DEBUG "value": [ 12:52:16 DEBUG "value": [ 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG [...] 12:52:16 DEBUG [...] 12:52:16 INFO [http_cmd]: http status code OK 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG --- status code --- 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG --- http response --- 12:52:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:52:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:52:16 DEBUG }, 12:52:16 DEBUG }, 12:52:16 DEBUG ----- output ----- 12:52:16 DEBUG "value": [ 12:52:16 DEBUG "value": [ 12:52:16 DEBUG { 12:52:16 DEBUG 1735908736, 12:52:16 DEBUG 1735908736, 12:52:16 DEBUG "status": "success", 12:52:16 DEBUG "14992310272" 12:52:16 DEBUG "758771712" 12:52:16 DEBUG "data": { 12:52:16 DEBUG ] 12:52:16 DEBUG ] 12:52:16 DEBUG "resultType": "vector", 12:52:16 DEBUG } 12:52:16 DEBUG } 12:52:16 DEBUG "result": [ 12:52:16 DEBUG ] 12:52:16 DEBUG ] 12:52:16 DEBUG { 12:52:16 DEBUG } 12:52:16 DEBUG } 12:52:16 DEBUG "metric": { 12:52:16 DEBUG } 12:52:16 DEBUG } 12:52:16 DEBUG "instance": "10.142.0.11:9100" 12:52:16 DEBUG }, 12:52:16 DEBUG "value": [ 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG [...] 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG 12:52:16 DEBUG "instance": "10.142.0.51:9100" 12:52:16 DEBUG }, 12:52:16 DEBUG "value": [ 12:52:16 DEBUG 1735908736, 12:52:16 DEBUG "0.0009999999999990906" 12:52:16 DEBUG ] 12:52:16 DEBUG } 12:52:16 DEBUG ] 12:52:16 DEBUG } 12:52:16 DEBUG } 12:52:22 INFO 12:52:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:52:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:52:22 INFO [loop_until]: OK (rc = 0) 12:52:22 DEBUG --- stdout --- 12:52:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 233m 1% 3863Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 197m 0% 19057Mi 67% 12:52:22 DEBUG --- stderr --- 12:52:22 DEBUG 12:52:26 INFO 12:52:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:52:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:52:26 INFO [loop_until]: OK (rc = 0) 12:52:26 DEBUG --- stdout --- 12:52:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 134m 13479Mi rcs-645c86fdd6-j4rsk 116m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:52:26 DEBUG --- stderr --- 12:52:26 DEBUG 12:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908794" 12:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908794" 12:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908794" 12:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908794" 12:53:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908794" 12:53:14 INFO [http_cmd]: http status code OK 12:53:14 DEBUG --- status code --- 12:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:14 DEBUG --- http response --- 12:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:14 ERROR [http_cmd]: ERROR 12:53:14 ERROR --- status code --- 12:53:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:53:14 ERROR --- http response --- 12:53:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:53:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:53:14 INFO [http_cmd]: http status code OK 12:53:14 DEBUG --- status code --- 12:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:14 DEBUG --- http response --- 12:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:14 INFO [http_cmd]: http status code OK 12:53:14 DEBUG --- status code --- 12:53:14 INFO [http_cmd]: http status code OK 12:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:14 DEBUG --- status code --- 12:53:14 DEBUG --- http response --- 12:53:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:14 DEBUG --- http response --- 12:53:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:53:15 DEBUG ----- output ----- 12:53:15 DEBUG { 12:53:15 DEBUG "status": "success", 12:53:15 DEBUG "data": { 12:53:15 DEBUG "resultType": "vector", 12:53:15 DEBUG "result": [ 12:53:15 DEBUG { 12:53:15 DEBUG "metric": { 12:53:15 DEBUG "pod": "ds-cts-0" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG [...] 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG "pod": "ds-idrepo-0" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 1735908795, 12:53:15 DEBUG "0" 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908795" 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:53:15 DEBUG ----- output ----- 12:53:15 DEBUG { 12:53:15 DEBUG "status": "success", 12:53:15 DEBUG "data": { 12:53:15 DEBUG "resultType": "vector", 12:53:15 DEBUG "result": [ 12:53:15 DEBUG { 12:53:15 DEBUG "metric": { 12:53:15 DEBUG "pod": "ds-cts-0" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG [...] 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG "pod": "ds-idrepo-0" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 1735908795, 12:53:15 DEBUG "0.005744797486617417" 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:53:15 DEBUG ----- output ----- 12:53:15 DEBUG { 12:53:15 DEBUG "status": "success", 12:53:15 DEBUG "data": { 12:53:15 DEBUG "resultType": "vector", 12:53:15 DEBUG "result": [ 12:53:15 DEBUG { 12:53:15 DEBUG "metric": { 12:53:15 DEBUG "instance": "10.142.0.11:9100" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG [...] 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG "instance": "10.142.0.51:9100" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 1735908795, 12:53:15 DEBUG "0.00039999999999812034" 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG } 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:53:15 DEBUG ----- output ----- 12:53:15 DEBUG { 12:53:15 DEBUG "status": "success", 12:53:15 DEBUG "data": { 12:53:15 DEBUG "resultType": "vector", 12:53:15 DEBUG "result": [ 12:53:15 DEBUG { 12:53:15 DEBUG "metric": { 12:53:15 DEBUG "pod": "ds-cts-0" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG [...] 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG "pod": "ds-idrepo-0" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 1735908795, 12:53:15 DEBUG "858.1005586592178" 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG } 12:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908795" 12:53:15 INFO [http_cmd]: http status code OK 12:53:15 DEBUG --- status code --- 12:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:15 DEBUG --- http response --- 12:53:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:53:15 DEBUG ----- output ----- 12:53:15 DEBUG { 12:53:15 DEBUG "status": "success", 12:53:15 DEBUG "data": { 12:53:15 DEBUG "resultType": "vector", 12:53:15 DEBUG "result": [ 12:53:15 DEBUG { 12:53:15 DEBUG "metric": { 12:53:15 DEBUG "instance": "10.142.0.11:9100" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG [...] 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG 12:53:15 DEBUG "instance": "10.142.0.51:9100" 12:53:15 DEBUG }, 12:53:15 DEBUG "value": [ 12:53:15 DEBUG 1735908795, 12:53:15 DEBUG "1228684.0666666667" 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG ] 12:53:15 DEBUG } 12:53:15 DEBUG } 12:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908796" 12:53:16 INFO [http_cmd]: http status code OK 12:53:16 DEBUG --- status code --- 12:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:16 DEBUG --- http response --- 12:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:53:16 DEBUG ----- output ----- 12:53:16 DEBUG { 12:53:16 DEBUG "status": "success", 12:53:16 DEBUG "data": { 12:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908796" 12:53:16 DEBUG "resultType": "vector", 12:53:16 DEBUG "result": [ 12:53:16 DEBUG { 12:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908796" 12:53:16 DEBUG "metric": { 12:53:16 DEBUG "instance": "10.142.0.11:9100" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG [...] 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG "instance": "10.142.0.51:9100" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 1735908796, 12:53:16 DEBUG "1245388.9666666666" 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG } 12:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908796" 12:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908796" 12:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908796" 12:53:16 INFO [http_cmd]: http status code OK 12:53:16 DEBUG --- status code --- 12:53:16 INFO [http_cmd]: http status code OK 12:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:16 DEBUG --- http response --- 12:53:16 DEBUG --- status code --- 12:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:53:16 DEBUG --- http response --- 12:53:16 DEBUG ----- output ----- 12:53:16 DEBUG { 12:53:16 DEBUG "status": "success", 12:53:16 DEBUG "data": { 12:53:16 DEBUG "resultType": "vector", 12:53:16 DEBUG "result": [ 12:53:16 DEBUG { 12:53:16 DEBUG "metric": { 12:53:16 DEBUG "instance": "10.142.0.11:9100" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG [...] 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG "instance": "10.142.0.51:9100" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 1735908796, 12:53:16 DEBUG "27716.266666666666" 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG } 12:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:53:16 DEBUG ----- output ----- 12:53:16 DEBUG { 12:53:16 DEBUG "status": "success", 12:53:16 DEBUG "data": { 12:53:16 DEBUG "resultType": "vector", 12:53:16 DEBUG "result": [ 12:53:16 DEBUG { 12:53:16 DEBUG "metric": { 12:53:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG [...] 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 1735908796, 12:53:16 DEBUG "0.03978073469227735" 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG } 12:53:16 INFO [http_cmd]: http status code OK 12:53:16 DEBUG --- status code --- 12:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:16 DEBUG --- http response --- 12:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:53:16 DEBUG ----- output ----- 12:53:16 DEBUG { 12:53:16 DEBUG "status": "success", 12:53:16 DEBUG "data": { 12:53:16 DEBUG "resultType": "vector", 12:53:16 DEBUG "result": [ 12:53:16 DEBUG { 12:53:16 DEBUG "metric": { 12:53:16 DEBUG "pod": "ds-idrepo-0" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG [...] 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 1735908796, 12:53:16 DEBUG "758771712" 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG } 12:53:16 INFO [http_cmd]: http status code OK 12:53:16 DEBUG --- status code --- 12:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:16 DEBUG --- http response --- 12:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:53:16 DEBUG ----- output ----- 12:53:16 DEBUG { 12:53:16 DEBUG "status": "success", 12:53:16 DEBUG "data": { 12:53:16 DEBUG "resultType": "vector", 12:53:16 DEBUG "result": [ 12:53:16 DEBUG { 12:53:16 DEBUG "metric": { 12:53:16 DEBUG "instance": "10.142.0.11:9100" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG [...] 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG "instance": "10.142.0.51:9100" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 1735908796, 12:53:16 DEBUG "0.0006666666666660604" 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG } 12:53:16 INFO [http_cmd]: http status code OK 12:53:16 DEBUG --- status code --- 12:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:16 DEBUG --- http response --- 12:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:53:16 DEBUG ----- output ----- 12:53:16 DEBUG { 12:53:16 DEBUG "status": "success", 12:53:16 DEBUG "data": { 12:53:16 DEBUG "resultType": "vector", 12:53:16 DEBUG "result": [ 12:53:16 DEBUG { 12:53:16 DEBUG "metric": { 12:53:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG [...] 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG 12:53:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:53:16 DEBUG }, 12:53:16 DEBUG "value": [ 12:53:16 DEBUG 1735908796, 12:53:16 DEBUG "14993190912" 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG ] 12:53:16 DEBUG } 12:53:16 DEBUG } 12:53:22 INFO 12:53:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:53:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:53:22 INFO [loop_until]: OK (rc = 0) 12:53:22 DEBUG --- stdout --- 12:53:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 202m 1% 3869Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 177m 0% 19058Mi 67% 12:53:22 DEBUG --- stderr --- 12:53:22 DEBUG 12:53:26 INFO 12:53:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:53:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:53:26 INFO [loop_until]: OK (rc = 0) 12:53:26 DEBUG --- stdout --- 12:53:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 83m 13479Mi rcs-645c86fdd6-j4rsk 86m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:53:26 DEBUG --- stderr --- 12:53:26 DEBUG 12:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908854" 12:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908854" 12:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908854" 12:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908854" 12:54:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908854" 12:54:14 INFO [http_cmd]: http status code OK 12:54:14 DEBUG --- status code --- 12:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:14 DEBUG --- http response --- 12:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:14 INFO [http_cmd]: http status code OK 12:54:14 INFO [http_cmd]: http status code OK 12:54:14 ERROR [http_cmd]: ERROR 12:54:14 INFO [http_cmd]: http status code OK 12:54:14 DEBUG --- status code --- 12:54:14 DEBUG --- status code --- 12:54:14 ERROR --- status code --- 12:54:14 DEBUG --- status code --- 12:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:54:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:14 DEBUG --- http response --- 12:54:14 DEBUG --- http response --- 12:54:14 ERROR --- http response --- 12:54:14 DEBUG --- http response --- 12:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:54:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:54:15 DEBUG ----- output ----- 12:54:15 DEBUG { 12:54:15 DEBUG "status": "success", 12:54:15 DEBUG "data": { 12:54:15 DEBUG "resultType": "vector", 12:54:15 DEBUG "result": [ 12:54:15 DEBUG { 12:54:15 DEBUG "metric": { 12:54:15 DEBUG "pod": "ds-cts-0" 12:54:15 DEBUG }, 12:54:15 DEBUG "value": [ 12:54:15 DEBUG 12:54:15 DEBUG 12:54:15 DEBUG 12:54:15 DEBUG [...] 12:54:15 DEBUG 12:54:15 DEBUG 12:54:15 DEBUG 12:54:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:54:15 DEBUG }, 12:54:15 DEBUG "value": [ 12:54:15 DEBUG 1735908855, 12:54:15 DEBUG "0" 12:54:15 DEBUG ] 12:54:15 DEBUG } 12:54:15 DEBUG ] 12:54:15 DEBUG } 12:54:15 DEBUG } 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908855" 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908855" 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:15 DEBUG ----- output ----- 12:54:15 DEBUG { 12:54:15 DEBUG "status": "success", 12:54:15 DEBUG "data": { 12:54:15 DEBUG "resultType": "vector", 12:54:15 DEBUG "result": [ 12:54:15 DEBUG { 12:54:15 DEBUG "metric": { 12:54:15 DEBUG "pod": "ds-cts-0" 12:54:15 DEBUG }, 12:54:15 DEBUG "value": [ 12:54:15 DEBUG 12:54:15 DEBUG 12:54:15 DEBUG 12:54:15 DEBUG [...] 12:54:15 DEBUG 12:54:15 DEBUG 12:54:15 DEBUG 12:54:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:54:15 DEBUG }, 12:54:15 DEBUG "value": [ 12:54:15 DEBUG 1735908855, 12:54:15 DEBUG "0" 12:54:15 DEBUG ] 12:54:15 DEBUG } 12:54:15 DEBUG ] 12:54:15 DEBUG } 12:54:15 DEBUG } 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:54:15 DEBUG ----- output ----- 12:54:15 DEBUG { 12:54:15 INFO [http_cmd]: http status code OK 12:54:15 DEBUG "status": "success", 12:54:15 DEBUG --- status code --- 12:54:15 DEBUG "data": { 12:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:15 DEBUG "resultType": "vector", 12:54:15 DEBUG --- http response --- 12:54:15 DEBUG "result": [ 12:54:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:54:15 DEBUG { 12:54:15 DEBUG ----- output ----- 12:54:15 DEBUG "metric": { 12:54:15 DEBUG { 12:54:15 DEBUG "instance": "10.142.0.11:9100" 12:54:15 DEBUG "status": "success", 12:54:15 DEBUG }, 12:54:15 DEBUG "data": { 12:54:15 DEBUG "value": [ 12:54:15 DEBUG "resultType": "vector", 12:54:15 DEBUG 12:54:15 DEBUG "result": [ 12:54:15 DEBUG 12:54:15 DEBUG { 12:54:15 DEBUG 12:54:15 DEBUG "metric": { 12:54:15 DEBUG [...] 12:54:15 DEBUG "pod": "ds-cts-0" 12:54:15 DEBUG 12:54:15 DEBUG }, 12:54:15 DEBUG 12:54:15 DEBUG "value": [ 12:54:15 DEBUG 12:54:15 DEBUG 12:54:15 DEBUG "instance": "10.142.0.51:9100" 12:54:15 DEBUG 12:54:15 DEBUG }, 12:54:15 DEBUG 12:54:15 DEBUG "value": [ 12:54:15 DEBUG [...] 12:54:15 DEBUG 1735908855, 12:54:15 DEBUG 12:54:15 DEBUG "0.0007000000000023951" 12:54:15 DEBUG 12:54:15 DEBUG ] 12:54:15 DEBUG 12:54:15 DEBUG } 12:54:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:54:15 DEBUG ] 12:54:15 DEBUG }, 12:54:15 DEBUG } 12:54:15 DEBUG "value": [ 12:54:15 DEBUG } 12:54:15 DEBUG 1735908855, 12:54:15 DEBUG "449.4305180047031" 12:54:15 DEBUG ] 12:54:15 DEBUG } 12:54:15 DEBUG ] 12:54:15 DEBUG } 12:54:15 DEBUG } 12:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908855" 12:54:16 INFO [http_cmd]: http status code OK 12:54:16 DEBUG --- status code --- 12:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:16 DEBUG --- http response --- 12:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:54:16 DEBUG ----- output ----- 12:54:16 DEBUG { 12:54:16 DEBUG "status": "success", 12:54:16 DEBUG "data": { 12:54:16 DEBUG "resultType": "vector", 12:54:16 DEBUG "result": [ 12:54:16 DEBUG { 12:54:16 DEBUG "metric": { 12:54:16 DEBUG "instance": "10.142.0.11:9100" 12:54:16 DEBUG }, 12:54:16 DEBUG "value": [ 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG [...] 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG "instance": "10.142.0.51:9100" 12:54:16 DEBUG }, 12:54:16 DEBUG "value": [ 12:54:16 DEBUG 1735908855, 12:54:16 DEBUG "617088.6666666665" 12:54:16 DEBUG ] 12:54:16 DEBUG } 12:54:16 DEBUG ] 12:54:16 DEBUG } 12:54:16 DEBUG } 12:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908856" 12:54:16 INFO [http_cmd]: http status code OK 12:54:16 DEBUG --- status code --- 12:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:16 DEBUG --- http response --- 12:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:54:16 DEBUG ----- output ----- 12:54:16 DEBUG { 12:54:16 DEBUG "status": "success", 12:54:16 DEBUG "data": { 12:54:16 DEBUG "resultType": "vector", 12:54:16 DEBUG "result": [ 12:54:16 DEBUG { 12:54:16 DEBUG "metric": { 12:54:16 DEBUG "instance": "10.142.0.11:9100" 12:54:16 DEBUG }, 12:54:16 DEBUG "value": [ 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG [...] 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG "instance": "10.142.0.51:9100" 12:54:16 DEBUG }, 12:54:16 DEBUG "value": [ 12:54:16 DEBUG 1735908856, 12:54:16 DEBUG "633925.3999999999" 12:54:16 DEBUG ] 12:54:16 DEBUG } 12:54:16 DEBUG ] 12:54:16 DEBUG } 12:54:16 DEBUG } 12:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908856" 12:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908856" 12:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908856" 12:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908856" 12:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908856" 12:54:16 INFO [http_cmd]: http status code OK 12:54:16 DEBUG --- status code --- 12:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:16 DEBUG --- http response --- 12:54:16 INFO [http_cmd]: http status code OK 12:54:16 DEBUG --- status code --- 12:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:54:16 DEBUG --- http response --- 12:54:16 DEBUG ----- output ----- 12:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:54:16 DEBUG { 12:54:16 DEBUG ----- output ----- 12:54:16 DEBUG "status": "success", 12:54:16 DEBUG { 12:54:16 DEBUG "data": { 12:54:16 DEBUG "status": "success", 12:54:16 DEBUG "resultType": "vector", 12:54:16 DEBUG "data": { 12:54:16 DEBUG "result": [ 12:54:16 DEBUG "resultType": "vector", 12:54:16 DEBUG { 12:54:16 DEBUG "result": [ 12:54:16 DEBUG "metric": { 12:54:16 DEBUG { 12:54:16 DEBUG "instance": "10.142.0.11:9100" 12:54:16 DEBUG "metric": { 12:54:16 DEBUG }, 12:54:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:54:16 DEBUG "value": [ 12:54:16 DEBUG }, 12:54:16 DEBUG 12:54:16 DEBUG "value": [ 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG [...] 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG [...] 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG "instance": "10.142.0.51:9100" 12:54:16 DEBUG 12:54:16 DEBUG }, 12:54:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:54:16 DEBUG "value": [ 12:54:16 DEBUG }, 12:54:16 DEBUG 1735908856, 12:54:16 DEBUG "value": [ 12:54:16 DEBUG "33177.600000000006" 12:54:16 DEBUG 1735908856, 12:54:16 DEBUG ] 12:54:16 DEBUG "0.048464345379111644" 12:54:16 DEBUG } 12:54:16 DEBUG ] 12:54:16 DEBUG ] 12:54:16 DEBUG } 12:54:16 DEBUG } 12:54:16 DEBUG ] 12:54:16 DEBUG } 12:54:16 DEBUG } 12:54:16 DEBUG } 12:54:16 INFO [http_cmd]: http status code OK 12:54:16 DEBUG --- status code --- 12:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:16 DEBUG --- http response --- 12:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:54:16 DEBUG ----- output ----- 12:54:16 DEBUG { 12:54:16 DEBUG "status": "success", 12:54:16 DEBUG "data": { 12:54:16 DEBUG "resultType": "vector", 12:54:16 DEBUG "result": [ 12:54:16 DEBUG { 12:54:16 DEBUG "metric": { 12:54:16 DEBUG "pod": "ds-idrepo-0" 12:54:16 DEBUG }, 12:54:16 DEBUG "value": [ 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG [...] 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:54:16 DEBUG }, 12:54:16 DEBUG "value": [ 12:54:16 DEBUG 1735908856, 12:54:16 DEBUG "758771712" 12:54:16 DEBUG ] 12:54:16 INFO [http_cmd]: http status code OK 12:54:16 DEBUG } 12:54:16 INFO [http_cmd]: http status code OK 12:54:16 DEBUG --- status code --- 12:54:16 DEBUG ] 12:54:16 DEBUG --- status code --- 12:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:16 DEBUG } 12:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:16 DEBUG --- http response --- 12:54:16 DEBUG } 12:54:16 DEBUG --- http response --- 12:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:54:16 DEBUG ----- output ----- 12:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:54:16 DEBUG { 12:54:16 DEBUG ----- output ----- 12:54:16 DEBUG "status": "success", 12:54:16 DEBUG { 12:54:16 DEBUG "data": { 12:54:16 DEBUG "status": "success", 12:54:16 DEBUG "resultType": "vector", 12:54:16 DEBUG "data": { 12:54:16 DEBUG "result": [ 12:54:16 DEBUG "resultType": "vector", 12:54:16 DEBUG { 12:54:16 DEBUG "result": [ 12:54:16 DEBUG "metric": { 12:54:16 DEBUG { 12:54:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:54:16 DEBUG "metric": { 12:54:16 DEBUG }, 12:54:16 DEBUG "instance": "10.142.0.11:9100" 12:54:16 DEBUG "value": [ 12:54:16 DEBUG }, 12:54:16 DEBUG 12:54:16 DEBUG "value": [ 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG [...] 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG [...] 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG 12:54:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:54:16 DEBUG 12:54:16 DEBUG }, 12:54:16 DEBUG "instance": "10.142.0.51:9100" 12:54:16 DEBUG "value": [ 12:54:16 DEBUG }, 12:54:16 DEBUG 1735908856, 12:54:16 DEBUG "value": [ 12:54:16 DEBUG "14991650816" 12:54:16 DEBUG 1735908856, 12:54:16 DEBUG ] 12:54:16 DEBUG "0.0003333333333330302" 12:54:16 DEBUG } 12:54:16 DEBUG ] 12:54:16 DEBUG ] 12:54:16 DEBUG } 12:54:16 DEBUG } 12:54:16 DEBUG ] 12:54:16 DEBUG } 12:54:16 DEBUG } 12:54:16 DEBUG } 12:54:22 INFO 12:54:22 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:54:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:54:23 INFO [loop_until]: OK (rc = 0) 12:54:23 DEBUG --- stdout --- 12:54:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 179m 1% 3862Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 131m 0% 19058Mi 67% 12:54:23 DEBUG --- stderr --- 12:54:23 DEBUG 12:54:26 INFO 12:54:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:54:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:54:26 INFO [loop_until]: OK (rc = 0) 12:54:26 DEBUG --- stdout --- 12:54:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 21m 13480Mi rcs-645c86fdd6-j4rsk 59m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:54:26 DEBUG --- stderr --- 12:54:26 DEBUG 12:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908914" 12:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908914" 12:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908914" 12:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908914" 12:55:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908914" 12:55:14 ERROR [http_cmd]: ERROR 12:55:14 INFO [http_cmd]: http status code OK 12:55:14 ERROR --- status code --- 12:55:14 INFO [http_cmd]: http status code OK 12:55:14 DEBUG --- status code --- 12:55:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:55:14 DEBUG --- status code --- 12:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:14 ERROR --- http response --- 12:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:14 INFO [http_cmd]: http status code OK 12:55:14 INFO [http_cmd]: http status code OK 12:55:14 DEBUG --- http response --- 12:55:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:55:14 DEBUG --- http response --- 12:55:14 DEBUG --- status code --- 12:55:14 DEBUG --- status code --- 12:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:14 DEBUG --- http response --- 12:55:14 DEBUG --- http response --- 12:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735908915, "0" ] } ] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908915" 12:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908915" 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735908915, "0.0006143760604894292" ] } ] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735908915, "252.7225050131112" ] } ] } } 12:55:15 INFO [http_cmd]: http status code OK 12:55:15 DEBUG --- status code --- 12:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:15 DEBUG --- http response --- 12:55:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:55:15 DEBUG ----- output ----- 12:55:15 DEBUG { 12:55:15 DEBUG "status": "success", 12:55:15 DEBUG "data": { 12:55:15 DEBUG "resultType": "vector", 12:55:15 DEBUG "result": [ 12:55:15 DEBUG { 12:55:15 DEBUG "metric": { 12:55:15 DEBUG "instance": "10.142.0.11:9100" 12:55:15 DEBUG }, 12:55:15 DEBUG "value": [ 12:55:15 DEBUG 12:55:15 DEBUG 12:55:15 DEBUG 12:55:15 DEBUG [...] 12:55:15 DEBUG 12:55:15 DEBUG 12:55:15 DEBUG 12:55:15 DEBUG "instance": "10.142.0.51:9100" 12:55:15 DEBUG }, 12:55:15 DEBUG "value": [ 12:55:15 DEBUG 1735908915, 12:55:15 DEBUG "0.0005333333333320904" 12:55:15 DEBUG ] 12:55:15 DEBUG } 12:55:15 DEBUG ] 12:55:15 DEBUG } 12:55:15 DEBUG } 12:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908916" 12:55:16 INFO [http_cmd]: http status code OK 12:55:16 DEBUG --- status code --- 12:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:16 DEBUG --- http response --- 12:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:55:16 DEBUG ----- output ----- 12:55:16 DEBUG { 12:55:16 DEBUG "status": "success", 12:55:16 DEBUG "data": { 12:55:16 DEBUG "resultType": "vector", 12:55:16 DEBUG "result": [ 12:55:16 DEBUG { 12:55:16 DEBUG "metric": { 12:55:16 DEBUG "instance": "10.142.0.11:9100" 12:55:16 DEBUG }, 12:55:16 DEBUG "value": [ 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG [...] 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG "instance": "10.142.0.51:9100" 12:55:16 DEBUG }, 12:55:16 DEBUG "value": [ 12:55:16 DEBUG 1735908916, 12:55:16 DEBUG "77639.8" 12:55:16 DEBUG ] 12:55:16 DEBUG } 12:55:16 DEBUG ] 12:55:16 DEBUG } 12:55:16 DEBUG } 12:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908916" 12:55:16 INFO [http_cmd]: http status code OK 12:55:16 DEBUG --- status code --- 12:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:16 DEBUG --- http response --- 12:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:55:16 DEBUG ----- output ----- 12:55:16 DEBUG { 12:55:16 DEBUG "status": "success", 12:55:16 DEBUG "data": { 12:55:16 DEBUG "resultType": "vector", 12:55:16 DEBUG "result": [ 12:55:16 DEBUG { 12:55:16 DEBUG "metric": { 12:55:16 DEBUG "instance": "10.142.0.11:9100" 12:55:16 DEBUG }, 12:55:16 DEBUG "value": [ 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG [...] 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG "instance": "10.142.0.51:9100" 12:55:16 DEBUG }, 12:55:16 DEBUG "value": [ 12:55:16 DEBUG 1735908916, 12:55:16 DEBUG "95310.2" 12:55:16 DEBUG ] 12:55:16 DEBUG } 12:55:16 DEBUG ] 12:55:16 DEBUG } 12:55:16 DEBUG } 12:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908916" 12:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908916" 12:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908916" 12:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908916" 12:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908916" 12:55:16 INFO [http_cmd]: http status code OK 12:55:16 DEBUG --- status code --- 12:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:16 DEBUG --- http response --- 12:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:55:16 DEBUG ----- output ----- 12:55:16 DEBUG { 12:55:16 DEBUG "status": "success", 12:55:16 DEBUG "data": { 12:55:16 DEBUG "resultType": "vector", 12:55:16 DEBUG "result": [ 12:55:16 DEBUG { 12:55:16 INFO [http_cmd]: http status code OK 12:55:16 DEBUG "metric": { 12:55:16 DEBUG --- status code --- 12:55:16 DEBUG "instance": "10.142.0.11:9100" 12:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:16 DEBUG }, 12:55:16 DEBUG --- http response --- 12:55:16 DEBUG "value": [ 12:55:16 DEBUG 12:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:55:16 DEBUG 12:55:16 DEBUG ----- output ----- 12:55:16 DEBUG 12:55:16 DEBUG { 12:55:16 DEBUG [...] 12:55:16 DEBUG "status": "success", 12:55:16 INFO [http_cmd]: http status code OK 12:55:16 DEBUG 12:55:16 INFO [http_cmd]: http status code OK 12:55:16 DEBUG "data": { 12:55:16 DEBUG --- status code --- 12:55:16 DEBUG 12:55:16 DEBUG --- status code --- 12:55:16 DEBUG "resultType": "vector", 12:55:16 DEBUG "result": [ 12:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:16 DEBUG 12:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:16 INFO [http_cmd]: http status code OK 12:55:16 DEBUG { 12:55:16 DEBUG --- http response --- 12:55:16 DEBUG "instance": "10.142.0.51:9100" 12:55:16 DEBUG --- http response --- 12:55:16 DEBUG --- status code --- 12:55:16 DEBUG "metric": { 12:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:55:16 DEBUG }, 12:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:55:16 DEBUG ----- output ----- 12:55:16 DEBUG "value": [ 12:55:16 DEBUG ----- output ----- 12:55:16 DEBUG --- http response --- 12:55:16 DEBUG }, 12:55:16 DEBUG { 12:55:16 DEBUG 1735908916, 12:55:16 DEBUG { 12:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:55:16 DEBUG "value": [ 12:55:16 DEBUG "status": "success", 12:55:16 DEBUG "31266.13333333333" 12:55:16 DEBUG "status": "success", 12:55:16 DEBUG ----- output ----- 12:55:16 DEBUG 12:55:16 DEBUG "data": { 12:55:16 DEBUG ] 12:55:16 DEBUG "data": { 12:55:16 DEBUG { 12:55:16 DEBUG 12:55:16 DEBUG "resultType": "vector", 12:55:16 DEBUG } 12:55:16 DEBUG "resultType": "vector", 12:55:16 DEBUG "status": "success", 12:55:16 DEBUG 12:55:16 DEBUG "result": [ 12:55:16 DEBUG ] 12:55:16 DEBUG "result": [ 12:55:16 DEBUG "data": { 12:55:16 DEBUG [...] 12:55:16 DEBUG { 12:55:16 DEBUG } 12:55:16 DEBUG { 12:55:16 DEBUG "resultType": "vector", 12:55:16 DEBUG 12:55:16 DEBUG "metric": { 12:55:16 DEBUG } 12:55:16 DEBUG "metric": { 12:55:16 DEBUG "result": [ 12:55:16 DEBUG 12:55:16 DEBUG "pod": "ds-idrepo-0" 12:55:16 DEBUG "instance": "10.142.0.11:9100" 12:55:16 DEBUG { 12:55:16 DEBUG 12:55:16 DEBUG }, 12:55:16 DEBUG }, 12:55:16 DEBUG "metric": { 12:55:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:55:16 DEBUG "value": [ 12:55:16 DEBUG "value": [ 12:55:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:55:16 DEBUG }, 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG }, 12:55:16 DEBUG "value": [ 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG "value": [ 12:55:16 DEBUG 1735908916, 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG "0.04716208724345687" 12:55:16 DEBUG [...] 12:55:16 DEBUG [...] 12:55:16 DEBUG 12:55:16 DEBUG ] 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG } 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG [...] 12:55:16 DEBUG ] 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG 12:55:16 DEBUG } 12:55:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:55:16 DEBUG "instance": "10.142.0.51:9100" 12:55:16 DEBUG 12:55:16 DEBUG } 12:55:16 DEBUG }, 12:55:16 DEBUG }, 12:55:16 DEBUG 12:55:16 DEBUG "value": [ 12:55:16 DEBUG "value": [ 12:55:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:55:16 DEBUG 1735908916, 12:55:16 DEBUG 1735908916, 12:55:16 DEBUG }, 12:55:16 DEBUG "758771712" 12:55:16 DEBUG "0.0003333333333330302" 12:55:16 DEBUG "value": [ 12:55:16 DEBUG ] 12:55:16 DEBUG ] 12:55:16 DEBUG 1735908916, 12:55:16 DEBUG } 12:55:16 DEBUG } 12:55:16 DEBUG "14991773696" 12:55:16 DEBUG ] 12:55:16 DEBUG ] 12:55:16 DEBUG ] 12:55:16 DEBUG } 12:55:16 DEBUG } 12:55:16 DEBUG } 12:55:16 DEBUG } 12:55:16 DEBUG } 12:55:16 DEBUG ] 12:55:16 DEBUG } 12:55:16 DEBUG } 12:55:23 INFO 12:55:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:55:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:55:23 INFO [loop_until]: OK (rc = 0) 12:55:23 DEBUG --- stdout --- 12:55:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 127m 0% 3866Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 86m 0% 19059Mi 67% 12:55:23 DEBUG --- stderr --- 12:55:23 DEBUG 12:55:26 INFO 12:55:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:55:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:55:26 INFO [loop_until]: OK (rc = 0) 12:55:26 DEBUG --- stdout --- 12:55:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 441m 13480Mi rcs-645c86fdd6-j4rsk 17m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:55:26 DEBUG --- stderr --- 12:55:26 DEBUG 12:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735908974" 12:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908974" 12:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908974" 12:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908974" 12:56:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908974" 12:56:14 INFO [http_cmd]: http status code OK 12:56:14 INFO [http_cmd]: http status code OK 12:56:14 INFO [http_cmd]: http status code OK 12:56:14 ERROR [http_cmd]: ERROR 12:56:14 DEBUG --- status code --- 12:56:14 DEBUG --- status code --- 12:56:14 DEBUG --- status code --- 12:56:14 ERROR --- status code --- 12:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:56:14 DEBUG --- http response --- 12:56:14 DEBUG --- http response --- 12:56:14 DEBUG --- http response --- 12:56:14 ERROR --- http response --- 12:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:56:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:56:14 INFO [http_cmd]: http status code OK 12:56:14 DEBUG --- status code --- 12:56:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:14 DEBUG --- http response --- 12:56:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG --- status code --- 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG --- status code --- 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735908975, "0" ] } ] } } 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735908975" 12:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908975" 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:56:15 DEBUG ----- output ----- 12:56:15 DEBUG { 12:56:15 DEBUG "status": "success", 12:56:15 DEBUG "data": { 12:56:15 DEBUG "resultType": "vector", 12:56:15 DEBUG "result": [ 12:56:15 DEBUG { 12:56:15 DEBUG "metric": { 12:56:15 DEBUG "pod": "ds-cts-0" 12:56:15 DEBUG }, 12:56:15 DEBUG "value": [ 12:56:15 DEBUG 12:56:15 DEBUG 12:56:15 DEBUG 12:56:15 DEBUG [...] 12:56:15 DEBUG 12:56:15 DEBUG 12:56:15 DEBUG 12:56:15 DEBUG "pod": "ds-idrepo-0" 12:56:15 DEBUG }, 12:56:15 DEBUG "value": [ 12:56:15 DEBUG 1735908975, 12:56:15 DEBUG "0" 12:56:15 DEBUG ] 12:56:15 DEBUG } 12:56:15 DEBUG ] 12:56:15 DEBUG } 12:56:15 DEBUG } 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735908975, "359.2153128518578" ] } ] } } 12:56:15 INFO [http_cmd]: http status code OK 12:56:15 DEBUG --- status code --- 12:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:15 DEBUG --- http response --- 12:56:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:56:15 DEBUG ----- output ----- 12:56:15 DEBUG { 12:56:15 DEBUG "status": "success", 12:56:15 DEBUG "data": { 12:56:15 DEBUG "resultType": "vector", 12:56:15 DEBUG "result": [ 12:56:15 DEBUG { 12:56:15 DEBUG "metric": { 12:56:15 DEBUG "instance": "10.142.0.11:9100" 12:56:15 DEBUG }, 12:56:15 DEBUG "value": [ 12:56:15 DEBUG 12:56:15 DEBUG 12:56:15 DEBUG 12:56:15 DEBUG [...] 12:56:15 DEBUG 12:56:15 DEBUG 12:56:15 DEBUG 12:56:15 DEBUG "instance": "10.142.0.51:9100" 12:56:15 DEBUG }, 12:56:15 DEBUG "value": [ 12:56:15 DEBUG 1735908975, 12:56:15 DEBUG "0.00036666666666936487" 12:56:15 DEBUG ] 12:56:15 DEBUG } 12:56:15 DEBUG ] 12:56:15 DEBUG } 12:56:15 DEBUG } 12:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908976" 12:56:16 INFO [http_cmd]: http status code OK 12:56:16 DEBUG --- status code --- 12:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:16 DEBUG --- http response --- 12:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:56:16 DEBUG ----- output ----- 12:56:16 DEBUG { 12:56:16 DEBUG "status": "success", 12:56:16 DEBUG "data": { 12:56:16 DEBUG "resultType": "vector", 12:56:16 DEBUG "result": [ 12:56:16 DEBUG { 12:56:16 DEBUG "metric": { 12:56:16 DEBUG "instance": "10.142.0.11:9100" 12:56:16 DEBUG }, 12:56:16 DEBUG "value": [ 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG [...] 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG "instance": "10.142.0.51:9100" 12:56:16 DEBUG }, 12:56:16 DEBUG "value": [ 12:56:16 DEBUG 1735908976, 12:56:16 DEBUG "73308.43333333335" 12:56:16 DEBUG ] 12:56:16 DEBUG } 12:56:16 DEBUG ] 12:56:16 DEBUG } 12:56:16 DEBUG } 12:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908976" 12:56:16 INFO [http_cmd]: http status code OK 12:56:16 DEBUG --- status code --- 12:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:16 DEBUG --- http response --- 12:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:56:16 DEBUG ----- output ----- 12:56:16 DEBUG { 12:56:16 DEBUG "status": "success", 12:56:16 DEBUG "data": { 12:56:16 DEBUG "resultType": "vector", 12:56:16 DEBUG "result": [ 12:56:16 DEBUG { 12:56:16 DEBUG "metric": { 12:56:16 DEBUG "instance": "10.142.0.11:9100" 12:56:16 DEBUG }, 12:56:16 DEBUG "value": [ 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG [...] 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG "instance": "10.142.0.51:9100" 12:56:16 DEBUG }, 12:56:16 DEBUG "value": [ 12:56:16 DEBUG 1735908976, 12:56:16 DEBUG "90192.63333333335" 12:56:16 DEBUG ] 12:56:16 DEBUG } 12:56:16 DEBUG ] 12:56:16 DEBUG } 12:56:16 DEBUG } 12:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908976" 12:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735908976" 12:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735908976" 12:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735908976" 12:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735908976" 12:56:16 INFO [http_cmd]: http status code OK 12:56:16 INFO [http_cmd]: http status code OK 12:56:16 DEBUG --- status code --- 12:56:16 DEBUG --- status code --- 12:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:16 DEBUG --- http response --- 12:56:16 DEBUG --- http response --- 12:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:56:16 DEBUG ----- output ----- 12:56:16 DEBUG ----- output ----- 12:56:16 DEBUG { 12:56:16 DEBUG { 12:56:16 DEBUG "status": "success", 12:56:16 DEBUG "status": "success", 12:56:16 DEBUG "data": { 12:56:16 DEBUG "data": { 12:56:16 DEBUG "resultType": "vector", 12:56:16 DEBUG "resultType": "vector", 12:56:16 DEBUG "result": [ 12:56:16 DEBUG "result": [ 12:56:16 DEBUG { 12:56:16 DEBUG { 12:56:16 DEBUG "metric": { 12:56:16 DEBUG "metric": { 12:56:16 DEBUG "instance": "10.142.0.11:9100" 12:56:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:56:16 DEBUG }, 12:56:16 DEBUG }, 12:56:16 DEBUG "value": [ 12:56:16 DEBUG "value": [ 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG [...] 12:56:16 DEBUG [...] 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 INFO [http_cmd]: http status code OK 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG --- status code --- 12:56:16 DEBUG 12:56:16 DEBUG "instance": "10.142.0.51:9100" 12:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:56:16 DEBUG }, 12:56:16 DEBUG --- http response --- 12:56:16 DEBUG }, 12:56:16 DEBUG "value": [ 12:56:16 DEBUG "value": [ 12:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:56:16 DEBUG 1735908976, 12:56:16 DEBUG 1735908976, 12:56:16 DEBUG ----- output ----- 12:56:16 DEBUG "13926.400000000001" 12:56:16 DEBUG "0.2174246232610796" 12:56:16 DEBUG { 12:56:16 DEBUG ] 12:56:16 DEBUG ] 12:56:16 DEBUG "status": "success", 12:56:16 DEBUG } 12:56:16 DEBUG } 12:56:16 DEBUG "data": { 12:56:16 DEBUG ] 12:56:16 DEBUG ] 12:56:16 DEBUG "resultType": "vector", 12:56:16 DEBUG } 12:56:16 DEBUG } 12:56:16 DEBUG "result": [ 12:56:16 DEBUG } 12:56:16 DEBUG } 12:56:16 DEBUG { 12:56:16 INFO [http_cmd]: http status code OK 12:56:16 DEBUG "metric": { 12:56:16 INFO [http_cmd]: http status code OK 12:56:16 DEBUG --- status code --- 12:56:16 DEBUG "instance": "10.142.0.11:9100" 12:56:16 DEBUG --- status code --- 12:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:16 DEBUG }, 12:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:16 DEBUG --- http response --- 12:56:16 DEBUG "value": [ 12:56:16 DEBUG --- http response --- 12:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:56:16 DEBUG 12:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:56:16 DEBUG ----- output ----- 12:56:16 DEBUG 12:56:16 DEBUG ----- output ----- 12:56:16 DEBUG { 12:56:16 DEBUG 12:56:16 DEBUG { 12:56:16 DEBUG "status": "success", 12:56:16 DEBUG [...] 12:56:16 DEBUG "status": "success", 12:56:16 DEBUG "data": { 12:56:16 DEBUG 12:56:16 DEBUG "data": { 12:56:16 DEBUG "resultType": "vector", 12:56:16 DEBUG 12:56:16 DEBUG "resultType": "vector", 12:56:16 DEBUG "result": [ 12:56:16 DEBUG 12:56:16 DEBUG "result": [ 12:56:16 DEBUG { 12:56:16 DEBUG "instance": "10.142.0.51:9100" 12:56:16 DEBUG { 12:56:16 DEBUG "metric": { 12:56:16 DEBUG }, 12:56:16 DEBUG "metric": { 12:56:16 DEBUG "pod": "ds-idrepo-0" 12:56:16 DEBUG "value": [ 12:56:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:56:16 DEBUG }, 12:56:16 DEBUG 1735908976, 12:56:16 DEBUG }, 12:56:16 DEBUG "value": [ 12:56:16 DEBUG "0.0003333333333330302" 12:56:16 DEBUG "value": [ 12:56:16 DEBUG 12:56:16 DEBUG ] 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG } 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG ] 12:56:16 DEBUG 12:56:16 DEBUG [...] 12:56:16 DEBUG } 12:56:16 DEBUG [...] 12:56:16 DEBUG 12:56:16 DEBUG } 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG 12:56:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:56:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:56:16 DEBUG }, 12:56:16 DEBUG }, 12:56:16 DEBUG "value": [ 12:56:16 DEBUG "value": [ 12:56:16 DEBUG 1735908976, 12:56:16 DEBUG 1735908976, 12:56:16 DEBUG "758779904" 12:56:16 DEBUG "14993166336" 12:56:16 DEBUG ] 12:56:16 DEBUG ] 12:56:16 DEBUG } 12:56:16 DEBUG } 12:56:16 DEBUG ] 12:56:16 DEBUG ] 12:56:16 DEBUG } 12:56:16 DEBUG } 12:56:16 DEBUG } 12:56:16 DEBUG } 12:56:23 INFO 12:56:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:56:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:56:23 INFO [loop_until]: OK (rc = 0) 12:56:23 DEBUG --- stdout --- 12:56:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 109m 0% 3869Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 76m 0% 19061Mi 67% 12:56:23 DEBUG --- stderr --- 12:56:23 DEBUG 12:56:26 INFO 12:56:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:56:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:56:26 INFO [loop_until]: OK (rc = 0) 12:56:26 DEBUG --- stdout --- 12:56:26 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 6m 13479Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:56:26 DEBUG --- stderr --- 12:56:26 DEBUG 12:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909034" 12:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909034" 12:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909034" 12:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909034" 12:57:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909034" 12:57:14 INFO [http_cmd]: http status code OK 12:57:14 DEBUG --- status code --- 12:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:14 DEBUG --- http response --- 12:57:14 ERROR [http_cmd]: ERROR 12:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:14 INFO [http_cmd]: http status code OK 12:57:14 ERROR --- status code --- 12:57:14 INFO [http_cmd]: http status code OK 12:57:14 DEBUG --- status code --- 12:57:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:57:14 DEBUG --- status code --- 12:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:14 INFO [http_cmd]: http status code OK 12:57:14 ERROR --- http response --- 12:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:14 DEBUG --- http response --- 12:57:14 DEBUG --- status code --- 12:57:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:57:14 DEBUG --- http response --- 12:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:14 DEBUG --- http response --- 12:57:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:15 DEBUG ----- output ----- 12:57:15 DEBUG { 12:57:15 DEBUG "status": "success", 12:57:15 DEBUG "data": { 12:57:15 DEBUG "resultType": "vector", 12:57:15 DEBUG "result": [ 12:57:15 DEBUG { 12:57:15 DEBUG "metric": { 12:57:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:57:15 DEBUG }, 12:57:15 DEBUG "value": [ 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG [...] 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG "pod": "ds-cts-0" 12:57:15 DEBUG }, 12:57:15 DEBUG "value": [ 12:57:15 DEBUG 1735909035, 12:57:15 DEBUG "0" 12:57:15 DEBUG ] 12:57:15 DEBUG } 12:57:15 DEBUG ] 12:57:15 DEBUG } 12:57:15 DEBUG } 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909035" 12:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909035" 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:15 DEBUG ----- output ----- 12:57:15 DEBUG { 12:57:15 DEBUG "status": "success", 12:57:15 DEBUG "data": { 12:57:15 DEBUG "resultType": "vector", 12:57:15 DEBUG "result": [ 12:57:15 DEBUG { 12:57:15 DEBUG "metric": { 12:57:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:57:15 DEBUG }, 12:57:15 DEBUG "value": [ 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG [...] 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG "pod": "ds-cts-0" 12:57:15 DEBUG }, 12:57:15 DEBUG "value": [ 12:57:15 DEBUG 1735909035, 12:57:15 DEBUG "0.0028402311995701737" 12:57:15 DEBUG ] 12:57:15 DEBUG } 12:57:15 DEBUG ] 12:57:15 DEBUG } 12:57:15 DEBUG } 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 INFO [http_cmd]: http status code OK 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG --- status code --- 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG --- http response --- 12:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:15 DEBUG ----- output ----- 12:57:15 DEBUG ----- output ----- 12:57:15 DEBUG { 12:57:15 DEBUG { 12:57:15 DEBUG "status": "success", 12:57:15 DEBUG "status": "success", 12:57:15 DEBUG "data": { 12:57:15 DEBUG "data": { 12:57:15 DEBUG "resultType": "vector", 12:57:15 DEBUG "resultType": "vector", 12:57:15 DEBUG "result": [ 12:57:15 DEBUG "result": [ 12:57:15 DEBUG { 12:57:15 DEBUG { 12:57:15 DEBUG "metric": { 12:57:15 DEBUG "metric": { 12:57:15 DEBUG "instance": "10.142.0.11:9100" 12:57:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:57:15 DEBUG }, 12:57:15 DEBUG }, 12:57:15 DEBUG "value": [ 12:57:15 DEBUG "value": [ 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG [...] 12:57:15 DEBUG [...] 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG 12:57:15 DEBUG "instance": "10.142.0.51:9100" 12:57:15 DEBUG "pod": "ds-cts-0" 12:57:15 DEBUG }, 12:57:15 DEBUG }, 12:57:15 DEBUG "value": [ 12:57:15 DEBUG "value": [ 12:57:15 DEBUG 1735909035, 12:57:15 DEBUG 1735909035, 12:57:15 DEBUG "0.0012333333333269063" 12:57:15 DEBUG "506.41362470250056" 12:57:15 DEBUG ] 12:57:15 DEBUG ] 12:57:15 DEBUG } 12:57:15 DEBUG } 12:57:15 DEBUG ] 12:57:15 DEBUG ] 12:57:15 DEBUG } 12:57:15 DEBUG } 12:57:15 DEBUG } 12:57:15 DEBUG } 12:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909036" 12:57:16 INFO [http_cmd]: http status code OK 12:57:16 DEBUG --- status code --- 12:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:16 DEBUG --- http response --- 12:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:16 DEBUG ----- output ----- 12:57:16 DEBUG { 12:57:16 DEBUG "status": "success", 12:57:16 DEBUG "data": { 12:57:16 DEBUG "resultType": "vector", 12:57:16 DEBUG "result": [ 12:57:16 DEBUG { 12:57:16 DEBUG "metric": { 12:57:16 DEBUG "instance": "10.142.0.11:9100" 12:57:16 DEBUG }, 12:57:16 DEBUG "value": [ 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG [...] 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG "instance": "10.142.0.51:9100" 12:57:16 DEBUG }, 12:57:16 DEBUG "value": [ 12:57:16 DEBUG 1735909036, 12:57:16 DEBUG "72816.36666666668" 12:57:16 DEBUG ] 12:57:16 DEBUG } 12:57:16 DEBUG ] 12:57:16 DEBUG } 12:57:16 DEBUG } 12:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909036" 12:57:16 INFO [http_cmd]: http status code OK 12:57:16 DEBUG --- status code --- 12:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:16 DEBUG --- http response --- 12:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:16 DEBUG ----- output ----- 12:57:16 DEBUG { 12:57:16 DEBUG "status": "success", 12:57:16 DEBUG "data": { 12:57:16 DEBUG "resultType": "vector", 12:57:16 DEBUG "result": [ 12:57:16 DEBUG { 12:57:16 DEBUG "metric": { 12:57:16 DEBUG "instance": "10.142.0.11:9100" 12:57:16 DEBUG }, 12:57:16 DEBUG "value": [ 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG [...] 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG "instance": "10.142.0.51:9100" 12:57:16 DEBUG }, 12:57:16 DEBUG "value": [ 12:57:16 DEBUG 1735909036, 12:57:16 DEBUG "89835.56666666667" 12:57:16 DEBUG ] 12:57:16 DEBUG } 12:57:16 DEBUG ] 12:57:16 DEBUG } 12:57:16 DEBUG } 12:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909036" 12:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909036" 12:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909036" 12:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909036" 12:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909036" 12:57:16 INFO [http_cmd]: http status code OK 12:57:16 DEBUG --- status code --- 12:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:16 INFO [http_cmd]: http status code OK 12:57:16 DEBUG --- http response --- 12:57:16 DEBUG --- status code --- 12:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:16 DEBUG ----- output ----- 12:57:16 DEBUG --- http response --- 12:57:16 DEBUG { 12:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:16 DEBUG "status": "success", 12:57:16 DEBUG ----- output ----- 12:57:16 DEBUG "data": { 12:57:16 DEBUG { 12:57:16 DEBUG "resultType": "vector", 12:57:16 DEBUG "status": "success", 12:57:16 DEBUG "result": [ 12:57:16 DEBUG "data": { 12:57:16 DEBUG { 12:57:16 DEBUG "resultType": "vector", 12:57:16 DEBUG "metric": { 12:57:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:57:16 DEBUG "result": [ 12:57:16 DEBUG }, 12:57:16 DEBUG { 12:57:16 INFO [http_cmd]: http status code OK 12:57:16 DEBUG "value": [ 12:57:16 DEBUG "metric": { 12:57:16 DEBUG --- status code --- 12:57:16 DEBUG 12:57:16 DEBUG "instance": "10.142.0.11:9100" 12:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:16 DEBUG 12:57:16 DEBUG }, 12:57:16 DEBUG --- http response --- 12:57:16 DEBUG 12:57:16 DEBUG "value": [ 12:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:16 DEBUG [...] 12:57:16 DEBUG 12:57:16 DEBUG ----- output ----- 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG { 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG "status": "success", 12:57:16 INFO [http_cmd]: http status code OK 12:57:16 DEBUG 12:57:16 DEBUG [...] 12:57:16 DEBUG "data": { 12:57:16 DEBUG --- status code --- 12:57:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:57:16 DEBUG 12:57:16 DEBUG "resultType": "vector", 12:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:16 DEBUG }, 12:57:16 DEBUG 12:57:16 INFO [http_cmd]: http status code OK 12:57:16 DEBUG "result": [ 12:57:16 DEBUG --- http response --- 12:57:16 DEBUG "value": [ 12:57:16 DEBUG 12:57:16 DEBUG --- status code --- 12:57:16 DEBUG { 12:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:57:16 DEBUG 1735909036, 12:57:16 DEBUG "instance": "10.142.0.51:9100" 12:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:16 DEBUG "metric": { 12:57:16 DEBUG ----- output ----- 12:57:16 DEBUG "0.14851509923780573" 12:57:16 DEBUG }, 12:57:16 DEBUG --- http response --- 12:57:16 DEBUG "instance": "10.142.0.11:9100" 12:57:16 DEBUG { 12:57:16 DEBUG ] 12:57:16 DEBUG "value": [ 12:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:57:16 DEBUG }, 12:57:16 DEBUG "status": "success", 12:57:16 DEBUG } 12:57:16 DEBUG 1735909036, 12:57:16 DEBUG ----- output ----- 12:57:16 DEBUG "value": [ 12:57:16 DEBUG "data": { 12:57:16 DEBUG ] 12:57:16 DEBUG "22937.6" 12:57:16 DEBUG { 12:57:16 DEBUG 12:57:16 DEBUG "resultType": "vector", 12:57:16 DEBUG } 12:57:16 DEBUG ] 12:57:16 DEBUG "status": "success", 12:57:16 DEBUG 12:57:16 DEBUG "result": [ 12:57:16 DEBUG } 12:57:16 DEBUG } 12:57:16 DEBUG "data": { 12:57:16 DEBUG 12:57:16 DEBUG { 12:57:16 DEBUG ] 12:57:16 DEBUG "resultType": "vector", 12:57:16 DEBUG [...] 12:57:16 DEBUG "metric": { 12:57:16 DEBUG } 12:57:16 DEBUG "result": [ 12:57:16 DEBUG 12:57:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:57:16 DEBUG } 12:57:16 DEBUG { 12:57:16 DEBUG 12:57:16 DEBUG }, 12:57:16 DEBUG "metric": { 12:57:16 DEBUG 12:57:16 DEBUG "value": [ 12:57:16 DEBUG "pod": "ds-idrepo-0" 12:57:16 DEBUG "instance": "10.142.0.51:9100" 12:57:16 DEBUG 12:57:16 DEBUG }, 12:57:16 DEBUG }, 12:57:16 DEBUG 12:57:16 DEBUG "value": [ 12:57:16 DEBUG "value": [ 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG 1735909036, 12:57:16 DEBUG [...] 12:57:16 DEBUG 12:57:16 DEBUG "0.001666666666666572" 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG ] 12:57:16 DEBUG 12:57:16 DEBUG 12:57:16 DEBUG } 12:57:16 DEBUG [...] 12:57:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:57:16 DEBUG ] 12:57:16 DEBUG 12:57:16 DEBUG }, 12:57:16 DEBUG } 12:57:16 DEBUG 12:57:16 DEBUG "value": [ 12:57:16 DEBUG } 12:57:16 DEBUG 12:57:16 DEBUG 1735909036, 12:57:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:57:16 DEBUG "14994194432" 12:57:16 DEBUG }, 12:57:16 DEBUG ] 12:57:16 DEBUG "value": [ 12:57:16 DEBUG } 12:57:16 DEBUG 1735909036, 12:57:16 DEBUG ] 12:57:16 DEBUG "758771712" 12:57:16 DEBUG } 12:57:16 DEBUG ] 12:57:16 DEBUG } 12:57:16 DEBUG } 12:57:16 DEBUG ] 12:57:16 DEBUG } 12:57:16 DEBUG } 12:57:23 INFO 12:57:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:57:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:57:23 INFO [loop_until]: OK (rc = 0) 12:57:23 DEBUG --- stdout --- 12:57:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 113m 0% 3862Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 75m 0% 19057Mi 67% 12:57:23 DEBUG --- stderr --- 12:57:23 DEBUG 12:57:26 INFO 12:57:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:57:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:57:27 INFO [loop_until]: OK (rc = 0) 12:57:27 DEBUG --- stdout --- 12:57:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13479Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:57:27 DEBUG --- stderr --- 12:57:27 DEBUG 12:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909094" 12:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909094" 12:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909094" 12:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909094" 12:58:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909094" 12:58:14 INFO [http_cmd]: http status code OK 12:58:14 DEBUG --- status code --- 12:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:14 DEBUG --- http response --- 12:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:14 ERROR [http_cmd]: ERROR 12:58:14 ERROR --- status code --- 12:58:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:58:14 ERROR --- http response --- 12:58:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:58:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:58:14 INFO [http_cmd]: http status code OK 12:58:14 DEBUG --- status code --- 12:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:14 INFO [http_cmd]: http status code OK 12:58:14 DEBUG --- http response --- 12:58:14 DEBUG --- status code --- 12:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:14 DEBUG --- http response --- 12:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:14 INFO [http_cmd]: http status code OK 12:58:14 DEBUG --- status code --- 12:58:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:14 DEBUG --- http response --- 12:58:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:15 DEBUG ----- output ----- 12:58:15 DEBUG { 12:58:15 DEBUG "status": "success", 12:58:15 DEBUG "data": { 12:58:15 DEBUG "resultType": "vector", 12:58:15 DEBUG "result": [ 12:58:15 DEBUG { 12:58:15 DEBUG "metric": { 12:58:15 DEBUG "pod": "ds-idrepo-0" 12:58:15 DEBUG }, 12:58:15 DEBUG "value": [ 12:58:15 DEBUG 12:58:15 DEBUG 12:58:15 DEBUG 12:58:15 DEBUG [...] 12:58:15 DEBUG 12:58:15 DEBUG 12:58:15 DEBUG 12:58:15 DEBUG "pod": "ds-cts-0" 12:58:15 DEBUG }, 12:58:15 DEBUG "value": [ 12:58:15 DEBUG 1735909095, 12:58:15 DEBUG "0" 12:58:15 DEBUG ] 12:58:15 DEBUG } 12:58:15 DEBUG ] 12:58:15 DEBUG } 12:58:15 DEBUG } 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909095" 12:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909095" 12:58:15 INFO [http_cmd]: http status code OK 12:58:15 DEBUG --- status code --- 12:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:15 DEBUG --- http response --- 12:58:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:58:15 DEBUG ----- output ----- 12:58:15 DEBUG { 12:58:15 DEBUG "status": "success", 12:58:15 DEBUG "data": { 12:58:15 DEBUG "resultType": "vector", 12:58:15 DEBUG "result": [ 12:58:15 DEBUG { 12:58:15 DEBUG "metric": { 12:58:15 DEBUG "pod": "ds-idrepo-0" 12:58:15 DEBUG }, 12:58:15 DEBUG "value": [ 12:58:15 DEBUG 12:58:15 DEBUG 12:58:15 DEBUG 12:58:15 DEBUG [...] 12:58:15 DEBUG 12:58:15 DEBUG 12:58:15 DEBUG 12:58:15 DEBUG "pod": "ds-cts-0" 12:58:15 DEBUG }, 12:58:15 DEBUG "value": [ 12:58:15 DEBUG 1735909095, 12:58:15 DEBUG "0.0027774755927513456" 12:58:15 DEBUG ] 12:58:15 DEBUG } 12:58:15 DEBUG ] 12:58:15 DEBUG } 12:58:15 DEBUG } 12:58:16 INFO [http_cmd]: http status code OK 12:58:16 DEBUG --- status code --- 12:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:16 INFO [http_cmd]: http status code OK 12:58:16 DEBUG --- http response --- 12:58:16 DEBUG --- status code --- 12:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:16 DEBUG ----- output ----- 12:58:16 DEBUG --- http response --- 12:58:16 DEBUG { 12:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:16 DEBUG "status": "success", 12:58:16 DEBUG ----- output ----- 12:58:16 DEBUG "data": { 12:58:16 DEBUG { 12:58:16 DEBUG "resultType": "vector", 12:58:16 DEBUG "status": "success", 12:58:16 DEBUG "result": [ 12:58:16 DEBUG "data": { 12:58:16 DEBUG { 12:58:16 DEBUG "resultType": "vector", 12:58:16 DEBUG "metric": { 12:58:16 DEBUG "result": [ 12:58:16 DEBUG "instance": "10.142.0.11:9100" 12:58:16 DEBUG { 12:58:16 DEBUG }, 12:58:16 DEBUG "metric": { 12:58:16 DEBUG "value": [ 12:58:16 DEBUG "pod": "ds-idrepo-0" 12:58:16 DEBUG 12:58:16 DEBUG }, 12:58:16 DEBUG 12:58:16 DEBUG "value": [ 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG [...] 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG [...] 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG "instance": "10.142.0.51:9100" 12:58:16 DEBUG 12:58:16 DEBUG }, 12:58:16 DEBUG 12:58:16 DEBUG "value": [ 12:58:16 DEBUG "pod": "ds-cts-0" 12:58:16 DEBUG 1735909095, 12:58:16 DEBUG }, 12:58:16 DEBUG "0.0003999999999981204" 12:58:16 DEBUG "value": [ 12:58:16 DEBUG ] 12:58:16 DEBUG 1735909095, 12:58:16 DEBUG } 12:58:16 DEBUG "428.45188284518827" 12:58:16 DEBUG ] 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909096" 12:58:16 INFO [http_cmd]: http status code OK 12:58:16 DEBUG --- status code --- 12:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:16 DEBUG --- http response --- 12:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:58:16 DEBUG ----- output ----- 12:58:16 DEBUG { 12:58:16 DEBUG "status": "success", 12:58:16 DEBUG "data": { 12:58:16 DEBUG "resultType": "vector", 12:58:16 DEBUG "result": [ 12:58:16 DEBUG { 12:58:16 DEBUG "metric": { 12:58:16 DEBUG "instance": "10.142.0.11:9100" 12:58:16 DEBUG }, 12:58:16 DEBUG "value": [ 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG [...] 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG "instance": "10.142.0.51:9100" 12:58:16 DEBUG }, 12:58:16 DEBUG "value": [ 12:58:16 DEBUG 1735909096, 12:58:16 DEBUG "791511.7666666667" 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909096" 12:58:16 INFO [http_cmd]: http status code OK 12:58:16 DEBUG --- status code --- 12:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:16 DEBUG --- http response --- 12:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:58:16 DEBUG ----- output ----- 12:58:16 DEBUG { 12:58:16 DEBUG "status": "success", 12:58:16 DEBUG "data": { 12:58:16 DEBUG "resultType": "vector", 12:58:16 DEBUG "result": [ 12:58:16 DEBUG { 12:58:16 DEBUG "metric": { 12:58:16 DEBUG "instance": "10.142.0.11:9100" 12:58:16 DEBUG }, 12:58:16 DEBUG "value": [ 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG [...] 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG "instance": "10.142.0.51:9100" 12:58:16 DEBUG }, 12:58:16 DEBUG "value": [ 12:58:16 DEBUG 1735909096, 12:58:16 DEBUG "806675.8000000003" 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909096" 12:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909096" 12:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909096" 12:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909096" 12:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909096" 12:58:16 INFO [http_cmd]: http status code OK 12:58:16 DEBUG --- status code --- 12:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:16 DEBUG --- http response --- 12:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:58:16 DEBUG ----- output ----- 12:58:16 DEBUG { 12:58:16 DEBUG "status": "success", 12:58:16 DEBUG "data": { 12:58:16 DEBUG "resultType": "vector", 12:58:16 DEBUG "result": [ 12:58:16 DEBUG { 12:58:16 DEBUG "metric": { 12:58:16 DEBUG "instance": "10.142.0.11:9100" 12:58:16 DEBUG }, 12:58:16 DEBUG "value": [ 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG [...] 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG "instance": "10.142.0.51:9100" 12:58:16 DEBUG }, 12:58:16 DEBUG "value": [ 12:58:16 DEBUG 1735909096, 12:58:16 DEBUG "19251.199999999997" 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:16 INFO [http_cmd]: http status code OK 12:58:16 DEBUG --- status code --- 12:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:16 DEBUG --- http response --- 12:58:16 INFO [http_cmd]: http status code OK 12:58:16 DEBUG --- status code --- 12:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:16 DEBUG --- http response --- 12:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:58:16 DEBUG ----- output ----- 12:58:16 DEBUG { 12:58:16 DEBUG "status": "success", 12:58:16 DEBUG "data": { 12:58:16 DEBUG "resultType": "vector", 12:58:16 DEBUG "result": [ 12:58:16 DEBUG { 12:58:16 DEBUG "metric": { 12:58:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:58:16 DEBUG }, 12:58:16 DEBUG "value": [ 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:58:16 DEBUG [...] 12:58:16 DEBUG ----- output ----- 12:58:16 DEBUG 12:58:16 DEBUG { 12:58:16 DEBUG 12:58:16 DEBUG "status": "success", 12:58:16 DEBUG 12:58:16 DEBUG "data": { 12:58:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:58:16 DEBUG "resultType": "vector", 12:58:16 DEBUG "result": [ 12:58:16 DEBUG { 12:58:16 DEBUG "metric": { 12:58:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:58:16 INFO [http_cmd]: http status code OK 12:58:16 DEBUG }, 12:58:16 INFO [http_cmd]: http status code OK 12:58:16 DEBUG }, 12:58:16 DEBUG --- status code --- 12:58:16 DEBUG "value": [ 12:58:16 DEBUG --- status code --- 12:58:16 DEBUG "value": [ 12:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:16 DEBUG 1735909096, 12:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:16 DEBUG 12:58:16 DEBUG --- http response --- 12:58:16 DEBUG "0.15969514037373508" 12:58:16 DEBUG --- http response --- 12:58:16 DEBUG 12:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:58:16 DEBUG ] 12:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:58:16 DEBUG 12:58:16 DEBUG ----- output ----- 12:58:16 DEBUG } 12:58:16 DEBUG ----- output ----- 12:58:16 DEBUG [...] 12:58:16 DEBUG { 12:58:16 DEBUG ] 12:58:16 DEBUG { 12:58:16 DEBUG 12:58:16 DEBUG "status": "success", 12:58:16 DEBUG } 12:58:16 DEBUG "status": "success", 12:58:16 DEBUG 12:58:16 DEBUG "data": { 12:58:16 DEBUG } 12:58:16 DEBUG "data": { 12:58:16 DEBUG 12:58:16 DEBUG "resultType": "vector", 12:58:16 DEBUG "resultType": "vector", 12:58:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:58:16 DEBUG "result": [ 12:58:16 DEBUG "result": [ 12:58:16 DEBUG }, 12:58:16 DEBUG { 12:58:16 DEBUG { 12:58:16 DEBUG "value": [ 12:58:16 DEBUG "metric": { 12:58:16 DEBUG "metric": { 12:58:16 DEBUG 1735909096, 12:58:16 DEBUG "pod": "ds-idrepo-0" 12:58:16 DEBUG "instance": "10.142.0.11:9100" 12:58:16 DEBUG "14989799424" 12:58:16 DEBUG }, 12:58:16 DEBUG }, 12:58:16 DEBUG ] 12:58:16 DEBUG "value": [ 12:58:16 DEBUG "value": [ 12:58:16 DEBUG } 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG ] 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG } 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG } 12:58:16 DEBUG [...] 12:58:16 DEBUG [...] 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG 12:58:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:58:16 DEBUG "instance": "10.142.0.51:9100" 12:58:16 DEBUG }, 12:58:16 DEBUG }, 12:58:16 DEBUG "value": [ 12:58:16 DEBUG "value": [ 12:58:16 DEBUG 1735909096, 12:58:16 DEBUG 1735909096, 12:58:16 DEBUG "758771712" 12:58:16 DEBUG "0" 12:58:16 DEBUG ] 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:16 DEBUG ] 12:58:16 DEBUG ] 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:16 DEBUG } 12:58:23 INFO 12:58:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:58:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:58:23 INFO [loop_until]: OK (rc = 0) 12:58:23 DEBUG --- stdout --- 12:58:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 111m 0% 3858Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 79m 0% 19055Mi 67% 12:58:23 DEBUG --- stderr --- 12:58:23 DEBUG 12:58:27 INFO 12:58:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:58:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:58:27 INFO [loop_until]: OK (rc = 0) 12:58:27 DEBUG --- stdout --- 12:58:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 6m 13478Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:58:27 DEBUG --- stderr --- 12:58:27 DEBUG 12:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909154" 12:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909154" 12:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909154" 12:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909154" 12:59:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909154" 12:59:14 INFO [http_cmd]: http status code OK 12:59:14 DEBUG --- status code --- 12:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:14 DEBUG --- http response --- 12:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:14 INFO [http_cmd]: http status code OK 12:59:14 DEBUG --- status code --- 12:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:14 DEBUG --- http response --- 12:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:14 ERROR [http_cmd]: ERROR 12:59:14 ERROR --- status code --- 12:59:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:59:14 ERROR --- http response --- 12:59:14 INFO [http_cmd]: http status code OK 12:59:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:59:14 DEBUG --- status code --- 12:59:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:14 INFO [http_cmd]: http status code OK 12:59:14 DEBUG --- http response --- 12:59:14 DEBUG --- status code --- 12:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:14 DEBUG --- http response --- 12:59:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:15 INFO [http_cmd]: http status code OK 12:59:15 DEBUG --- status code --- 12:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:15 DEBUG --- http response --- 12:59:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:15 DEBUG ----- output ----- 12:59:15 DEBUG { 12:59:15 DEBUG "status": "success", 12:59:15 DEBUG "data": { 12:59:15 DEBUG "resultType": "vector", 12:59:15 DEBUG "result": [ 12:59:15 DEBUG { 12:59:15 DEBUG "metric": { 12:59:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:59:15 DEBUG }, 12:59:15 DEBUG "value": [ 12:59:15 DEBUG 12:59:15 DEBUG 12:59:15 DEBUG 12:59:15 DEBUG [...] 12:59:15 DEBUG 12:59:15 DEBUG 12:59:15 DEBUG 12:59:15 DEBUG "pod": "ds-idrepo-0" 12:59:15 DEBUG }, 12:59:15 DEBUG "value": [ 12:59:15 DEBUG 1735909155, 12:59:15 DEBUG "2045825.2365878725" 12:59:15 DEBUG ] 12:59:15 DEBUG } 12:59:15 DEBUG ] 12:59:15 DEBUG } 12:59:15 DEBUG } 12:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909155" 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG { 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG "data": { 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG "result": [ 12:59:16 DEBUG { 12:59:16 DEBUG "metric": { 12:59:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG [...] 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG "pod": "ds-idrepo-0" 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 1735909155, 12:59:16 DEBUG "0.09772531577605818" 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909156" 12:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909156" 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG { 12:59:16 DEBUG { 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG "data": { 12:59:16 DEBUG "data": { 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG "result": [ 12:59:16 DEBUG "result": [ 12:59:16 DEBUG { 12:59:16 DEBUG { 12:59:16 DEBUG "metric": { 12:59:16 DEBUG "metric": { 12:59:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 12:59:16 DEBUG "instance": "10.142.0.11:9100" 12:59:16 DEBUG }, 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG [...] 12:59:16 DEBUG [...] 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG "pod": "ds-idrepo-0" 12:59:16 DEBUG "instance": "10.142.0.51:9100" 12:59:16 DEBUG }, 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 1735909156, 12:59:16 DEBUG 1735909156, 12:59:16 DEBUG "208.41329222336418" 12:59:16 DEBUG "0.00040000000000190995" 12:59:16 DEBUG ] 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:16 DEBUG ] 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909156" 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG { 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG "data": { 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG "result": [ 12:59:16 DEBUG { 12:59:16 DEBUG "metric": { 12:59:16 DEBUG "instance": "10.142.0.11:9100" 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG [...] 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG "instance": "10.142.0.51:9100" 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 1735909156, 12:59:16 DEBUG "1229503.6" 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909156" 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG { 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG "data": { 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG "result": [ 12:59:16 DEBUG { 12:59:16 DEBUG "metric": { 12:59:16 DEBUG "instance": "10.142.0.11:9100" 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG [...] 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG "instance": "10.142.0.51:9100" 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 1735909156, 12:59:16 DEBUG "1246896.8" 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909156" 12:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909156" 12:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909156" 12:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909156" 12:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909156" 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG { 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG "data": { 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG "result": [ 12:59:16 DEBUG { 12:59:16 DEBUG "metric": { 12:59:16 DEBUG "instance": "10.142.0.11:9100" 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG [...] 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG "instance": "10.142.0.51:9100" 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 1735909156, 12:59:16 DEBUG "23210.666666666664" 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG { 12:59:16 DEBUG { 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG "data": { 12:59:16 DEBUG "data": { 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG "result": [ 12:59:16 DEBUG "result": [ 12:59:16 DEBUG { 12:59:16 DEBUG "metric": { 12:59:16 DEBUG { 12:59:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 DEBUG "metric": { 12:59:16 DEBUG }, 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG "instance": "10.142.0.11:9100" 12:59:16 DEBUG 12:59:16 INFO [http_cmd]: http status code OK 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG }, 12:59:16 DEBUG 12:59:16 DEBUG [...] 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG "value": [ 12:59:16 DEBUG --- status code --- 12:59:16 DEBUG 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 12:59:16 DEBUG 12:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:16 DEBUG 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG 12:59:16 DEBUG --- http response --- 12:59:16 DEBUG 12:59:16 DEBUG { 12:59:16 DEBUG 12:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 12:59:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG [...] 12:59:16 DEBUG ----- output ----- 12:59:16 DEBUG }, 12:59:16 DEBUG "data": { 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG "value": [ 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG { 12:59:16 DEBUG 12:59:16 DEBUG 1735909156, 12:59:16 DEBUG "result": [ 12:59:16 DEBUG "status": "success", 12:59:16 DEBUG "instance": "10.142.0.51:9100" 12:59:16 DEBUG "0.18565317169396933" 12:59:16 DEBUG { 12:59:16 DEBUG "data": { 12:59:16 DEBUG }, 12:59:16 DEBUG ] 12:59:16 DEBUG "metric": { 12:59:16 DEBUG "resultType": "vector", 12:59:16 DEBUG "value": [ 12:59:16 DEBUG } 12:59:16 DEBUG "pod": "ds-idrepo-0" 12:59:16 DEBUG "result": [ 12:59:16 DEBUG 1735909156, 12:59:16 DEBUG ] 12:59:16 DEBUG }, 12:59:16 DEBUG { 12:59:16 DEBUG "0.0003333333333330302" 12:59:16 DEBUG } 12:59:16 DEBUG "value": [ 12:59:16 DEBUG "metric": { 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG 12:59:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 12:59:16 DEBUG } 12:59:16 DEBUG 12:59:16 DEBUG }, 12:59:16 DEBUG ] 12:59:16 DEBUG 12:59:16 DEBUG "value": [ 12:59:16 DEBUG } 12:59:16 DEBUG [...] 12:59:16 DEBUG 12:59:16 DEBUG } 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG 12:59:16 DEBUG [...] 12:59:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 12:59:16 DEBUG 12:59:16 DEBUG }, 12:59:16 DEBUG 12:59:16 DEBUG "value": [ 12:59:16 DEBUG 12:59:16 DEBUG 1735909156, 12:59:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 12:59:16 DEBUG "758771712" 12:59:16 DEBUG }, 12:59:16 DEBUG ] 12:59:16 DEBUG "value": [ 12:59:16 DEBUG } 12:59:16 DEBUG 1735909156, 12:59:16 DEBUG ] 12:59:16 DEBUG "14990544896" 12:59:16 DEBUG } 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:16 DEBUG ] 12:59:16 DEBUG } 12:59:16 DEBUG } 12:59:23 INFO 12:59:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 12:59:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:59:23 INFO [loop_until]: OK (rc = 0) 12:59:23 DEBUG --- stdout --- 12:59:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 154m 0% 3850Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 157m 0% 19087Mi 67% 12:59:23 DEBUG --- stderr --- 12:59:23 DEBUG 12:59:27 INFO 12:59:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 12:59:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:59:27 INFO [loop_until]: OK (rc = 0) 12:59:27 DEBUG --- stdout --- 12:59:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 93m 13479Mi rcs-645c86fdd6-j4rsk 68m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 12:59:27 DEBUG --- stderr --- 12:59:27 DEBUG 13:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909214" 13:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909214" 13:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909214" 13:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909214" 13:00:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909214" 13:00:14 ERROR [http_cmd]: ERROR 13:00:14 ERROR --- status code --- 13:00:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:00:14 ERROR --- http response --- 13:00:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:00:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:00:14 INFO [http_cmd]: http status code OK 13:00:14 DEBUG --- status code --- 13:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:14 DEBUG --- http response --- 13:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:14 INFO [http_cmd]: http status code OK 13:00:14 DEBUG --- status code --- 13:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:14 DEBUG --- http response --- 13:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:14 INFO [http_cmd]: http status code OK 13:00:14 DEBUG --- status code --- 13:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:14 INFO [http_cmd]: http status code OK 13:00:14 DEBUG --- http response --- 13:00:14 DEBUG --- status code --- 13:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:14 DEBUG --- http response --- 13:00:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909215" 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909215" 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG --- status code --- 13:00:15 INFO [http_cmd]: http status code OK 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG --- status code --- 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:15 DEBUG --- http response --- 13:00:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:15 DEBUG ----- output ----- 13:00:15 DEBUG { 13:00:15 DEBUG "status": "success", 13:00:15 DEBUG "data": { 13:00:15 DEBUG "resultType": "vector", 13:00:15 DEBUG "result": [ 13:00:15 DEBUG { 13:00:15 DEBUG "metric": { 13:00:15 DEBUG "pod": "ds-cts-0" 13:00:15 DEBUG }, 13:00:15 DEBUG "value": [ 13:00:15 DEBUG 13:00:15 DEBUG 13:00:15 DEBUG 13:00:15 DEBUG [...] 13:00:15 DEBUG 13:00:15 DEBUG 13:00:15 DEBUG 13:00:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:00:15 DEBUG }, 13:00:15 DEBUG "value": [ 13:00:15 DEBUG 1735909215, 13:00:15 DEBUG "0" 13:00:15 DEBUG ] 13:00:15 DEBUG } 13:00:15 DEBUG ] 13:00:15 DEBUG } 13:00:15 DEBUG } 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909216" 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG { 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG "data": { 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "result": [ 13:00:16 DEBUG { 13:00:16 DEBUG "metric": { 13:00:16 DEBUG "pod": "ds-cts-0" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG [...] 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG "0" 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG } 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909216" 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909216" 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG { 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG "data": { 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "result": [ 13:00:16 DEBUG { 13:00:16 DEBUG "metric": { 13:00:16 DEBUG "pod": "ds-cts-0" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG [...] 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG "0" 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG } 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG { 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG "data": { 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "result": [ 13:00:16 DEBUG { 13:00:16 DEBUG "metric": { 13:00:16 DEBUG "instance": "10.142.0.11:9100" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG [...] 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG "instance": "10.142.0.51:9100" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG "0.00040000000000190995" 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG } 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909216" 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG { 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG "data": { 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "result": [ 13:00:16 DEBUG { 13:00:16 DEBUG "metric": { 13:00:16 DEBUG "instance": "10.142.0.11:9100" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG [...] 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG "instance": "10.142.0.51:9100" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG "1231802.7666666666" 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG } 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909216" 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG { 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG "data": { 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "result": [ 13:00:16 DEBUG { 13:00:16 DEBUG "metric": { 13:00:16 DEBUG "instance": "10.142.0.11:9100" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG [...] 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG "instance": "10.142.0.51:9100" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG "1248612.0666666667" 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG } 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909216" 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909216" 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909216" 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG { 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG "data": { 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "result": [ 13:00:16 DEBUG { 13:00:16 DEBUG "metric": { 13:00:16 DEBUG "instance": "10.142.0.11:9100" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG [...] 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG "instance": "10.142.0.51:9100" 13:00:16 DEBUG }, 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG "21572.266666666666" 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG } 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909216" 13:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909216" 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG { 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG "data": { 13:00:16 DEBUG { 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG "result": [ 13:00:16 DEBUG "data": { 13:00:16 DEBUG { 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "metric": { 13:00:16 DEBUG "result": [ 13:00:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:00:16 DEBUG { 13:00:16 DEBUG }, 13:00:16 DEBUG "metric": { 13:00:16 DEBUG "value": [ 13:00:16 DEBUG "instance": "10.142.0.11:9100" 13:00:16 DEBUG 13:00:16 DEBUG }, 13:00:16 DEBUG 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG [...] 13:00:16 DEBUG 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG 13:00:16 DEBUG [...] 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:00:16 DEBUG 13:00:16 INFO [http_cmd]: http status code OK 13:00:16 DEBUG }, 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:00:16 DEBUG 13:00:16 DEBUG --- status code --- 13:00:16 DEBUG "value": [ 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG "instance": "10.142.0.51:9100" 13:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG { 13:00:16 DEBUG }, 13:00:16 DEBUG --- http response --- 13:00:16 DEBUG "0.1176110187717317" 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG "value": [ 13:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:00:16 DEBUG ] 13:00:16 DEBUG "data": { 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG ----- output ----- 13:00:16 DEBUG } 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "0.0003333333333330302" 13:00:16 DEBUG { 13:00:16 DEBUG ] 13:00:16 DEBUG "result": [ 13:00:16 DEBUG ] 13:00:16 DEBUG "status": "success", 13:00:16 DEBUG } 13:00:16 DEBUG { 13:00:16 DEBUG } 13:00:16 DEBUG "data": { 13:00:16 DEBUG } 13:00:16 DEBUG "metric": { 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG } 13:00:16 DEBUG "resultType": "vector", 13:00:16 DEBUG "pod": "ds-idrepo-0" 13:00:16 DEBUG "result": [ 13:00:16 DEBUG }, 13:00:16 DEBUG { 13:00:16 DEBUG "value": [ 13:00:16 DEBUG "metric": { 13:00:16 DEBUG 13:00:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:00:16 DEBUG 13:00:16 DEBUG }, 13:00:16 DEBUG 13:00:16 DEBUG "value": [ 13:00:16 DEBUG [...] 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG 13:00:16 DEBUG [...] 13:00:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:00:16 DEBUG 13:00:16 DEBUG }, 13:00:16 DEBUG 13:00:16 DEBUG "value": [ 13:00:16 DEBUG 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:00:16 DEBUG "758771712" 13:00:16 DEBUG }, 13:00:16 DEBUG ] 13:00:16 DEBUG "value": [ 13:00:16 DEBUG } 13:00:16 DEBUG 1735909216, 13:00:16 DEBUG ] 13:00:16 DEBUG "15020007424" 13:00:16 DEBUG } 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG } 13:00:16 DEBUG ] 13:00:16 DEBUG } 13:00:16 DEBUG } 13:00:23 INFO 13:00:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:00:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:00:23 INFO [loop_until]: OK (rc = 0) 13:00:23 DEBUG --- stdout --- 13:00:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 202m 1% 3853Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 126m 0% 19085Mi 67% 13:00:23 DEBUG --- stderr --- 13:00:23 DEBUG 13:00:27 INFO 13:00:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:00:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:00:27 INFO [loop_until]: OK (rc = 0) 13:00:27 DEBUG --- stdout --- 13:00:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 72m 13479Mi rcs-645c86fdd6-j4rsk 56m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:00:27 DEBUG --- stderr --- 13:00:27 DEBUG 13:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909274" 13:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909274" 13:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909274" 13:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909274" 13:01:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909274" 13:01:14 ERROR [http_cmd]: ERROR 13:01:14 INFO [http_cmd]: http status code OK 13:01:14 ERROR --- status code --- 13:01:14 INFO [http_cmd]: http status code OK 13:01:14 DEBUG --- status code --- 13:01:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:01:14 DEBUG --- status code --- 13:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:14 ERROR --- http response --- 13:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:14 DEBUG --- http response --- 13:01:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:01:14 DEBUG --- http response --- 13:01:14 INFO [http_cmd]: http status code OK 13:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:14 DEBUG --- status code --- 13:01:14 INFO [http_cmd]: http status code OK 13:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:14 DEBUG --- status code --- 13:01:14 DEBUG --- http response --- 13:01:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:14 DEBUG --- http response --- 13:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909275" 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:15 INFO [http_cmd]: http status code OK 13:01:15 DEBUG --- status code --- 13:01:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:15 DEBUG --- http response --- 13:01:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:01:15 DEBUG ----- output ----- 13:01:15 DEBUG { 13:01:15 DEBUG "status": "success", 13:01:15 DEBUG "data": { 13:01:15 DEBUG "resultType": "vector", 13:01:15 DEBUG "result": [ 13:01:15 DEBUG { 13:01:15 DEBUG "metric": { 13:01:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:01:15 DEBUG }, 13:01:15 DEBUG "value": [ 13:01:15 DEBUG 13:01:15 DEBUG 13:01:15 DEBUG 13:01:15 DEBUG [...] 13:01:15 DEBUG 13:01:15 DEBUG 13:01:15 DEBUG 13:01:15 DEBUG "pod": "ds-cts-0" 13:01:15 DEBUG }, 13:01:15 DEBUG "value": [ 13:01:15 DEBUG 1735909275, 13:01:15 DEBUG "0" 13:01:15 DEBUG ] 13:01:15 DEBUG } 13:01:15 DEBUG ] 13:01:15 DEBUG } 13:01:15 DEBUG } 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909276" 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG --- http response --- 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG { 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG "data": { 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG "result": [ 13:01:16 DEBUG { 13:01:16 DEBUG "metric": { 13:01:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG [...] 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG "pod": "ds-cts-0" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG "0.002808019611277662" 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909276" 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909276" 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG --- http response --- 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG { 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG "data": { 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG "result": [ 13:01:16 DEBUG { 13:01:16 DEBUG "metric": { 13:01:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG [...] 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG "pod": "ds-cts-0" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG "430.32743827700926" 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG --- http response --- 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG { 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG "data": { 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG "result": [ 13:01:16 DEBUG { 13:01:16 DEBUG "metric": { 13:01:16 DEBUG "instance": "10.142.0.11:9100" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG [...] 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG "instance": "10.142.0.51:9100" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG "0.0007999999999962407" 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909276" 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG --- http response --- 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG { 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG "data": { 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG "result": [ 13:01:16 DEBUG { 13:01:16 DEBUG "metric": { 13:01:16 DEBUG "instance": "10.142.0.11:9100" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG [...] 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG "instance": "10.142.0.51:9100" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG "2051557.8666666665" 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909276" 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG --- http response --- 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG { 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG "data": { 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG "result": [ 13:01:16 DEBUG { 13:01:16 DEBUG "metric": { 13:01:16 DEBUG "instance": "10.142.0.11:9100" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG [...] 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG "instance": "10.142.0.51:9100" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG "2068909.6999999995" 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909276" 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG --- http response --- 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG { 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG "data": { 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG "result": [ 13:01:16 DEBUG { 13:01:16 DEBUG "metric": { 13:01:16 DEBUG "instance": "10.142.0.11:9100" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG [...] 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG "instance": "10.142.0.51:9100" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG "161245.86666666667" 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909276" 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909276" 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909276" 13:01:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909276" 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG --- http response --- 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG { 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG "data": { 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG "result": [ 13:01:16 DEBUG { 13:01:16 DEBUG "metric": { 13:01:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG [...] 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:01:16 DEBUG }, 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG "0.5738464238451476" 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG --- http response --- 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG --- http response --- 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG { 13:01:16 INFO [http_cmd]: http status code OK 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:01:16 DEBUG --- status code --- 13:01:16 DEBUG "data": { 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG { 13:01:16 DEBUG --- http response --- 13:01:16 DEBUG "result": [ 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:01:16 DEBUG { 13:01:16 DEBUG "data": { 13:01:16 DEBUG ----- output ----- 13:01:16 DEBUG "metric": { 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG { 13:01:16 DEBUG "pod": "ds-idrepo-0" 13:01:16 DEBUG "result": [ 13:01:16 DEBUG "status": "success", 13:01:16 DEBUG }, 13:01:16 DEBUG { 13:01:16 DEBUG "data": { 13:01:16 DEBUG "value": [ 13:01:16 DEBUG "metric": { 13:01:16 DEBUG "resultType": "vector", 13:01:16 DEBUG 13:01:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:01:16 DEBUG "result": [ 13:01:16 DEBUG 13:01:16 DEBUG }, 13:01:16 DEBUG { 13:01:16 DEBUG 13:01:16 DEBUG "value": [ 13:01:16 DEBUG "metric": { 13:01:16 DEBUG [...] 13:01:16 DEBUG 13:01:16 DEBUG "instance": "10.142.0.11:9100" 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG }, 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 13:01:16 DEBUG [...] 13:01:16 DEBUG 13:01:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG }, 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG "value": [ 13:01:16 DEBUG 13:01:16 DEBUG [...] 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:01:16 DEBUG 13:01:16 DEBUG 13:01:16 DEBUG }, 13:01:16 DEBUG "758771712" 13:01:16 DEBUG 13:01:16 DEBUG "value": [ 13:01:16 DEBUG ] 13:01:16 DEBUG "instance": "10.142.0.51:9100" 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG } 13:01:16 DEBUG }, 13:01:16 DEBUG "14996500480" 13:01:16 DEBUG ] 13:01:16 DEBUG "value": [ 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG 1735909276, 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 DEBUG "0.0003333333333330302" 13:01:16 DEBUG ] 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:16 DEBUG ] 13:01:16 DEBUG } 13:01:16 DEBUG } 13:01:23 INFO 13:01:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:01:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:01:23 INFO [loop_until]: OK (rc = 0) 13:01:23 DEBUG --- stdout --- 13:01:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 170m 1% 3817Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 674m 2% 19062Mi 67% 13:01:23 DEBUG --- stderr --- 13:01:23 DEBUG 13:01:27 INFO 13:01:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:01:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:01:27 INFO [loop_until]: OK (rc = 0) 13:01:27 DEBUG --- stdout --- 13:01:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 695m 13480Mi rcs-645c86fdd6-j4rsk 133m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:01:27 DEBUG --- stderr --- 13:01:27 DEBUG 13:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909334" 13:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909334" 13:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909334" 13:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909334" 13:02:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909334" 13:02:14 INFO [http_cmd]: http status code OK 13:02:14 DEBUG --- status code --- 13:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:14 DEBUG --- http response --- 13:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:14 ERROR [http_cmd]: ERROR 13:02:14 ERROR --- status code --- 13:02:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:02:14 INFO [http_cmd]: http status code OK 13:02:14 INFO [http_cmd]: http status code OK 13:02:14 ERROR --- http response --- 13:02:14 INFO [http_cmd]: http status code OK 13:02:14 DEBUG --- status code --- 13:02:14 DEBUG --- status code --- 13:02:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:02:14 DEBUG --- status code --- 13:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:02:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:14 DEBUG --- http response --- 13:02:14 DEBUG --- http response --- 13:02:14 DEBUG --- http response --- 13:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909335" 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 INFO [http_cmd]: http status code OK 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG --- status code --- 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG --- http response --- 13:02:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:15 DEBUG ----- output ----- 13:02:15 DEBUG { 13:02:15 DEBUG "status": "success", 13:02:15 DEBUG "data": { 13:02:15 DEBUG "resultType": "vector", 13:02:15 DEBUG "result": [ 13:02:15 DEBUG { 13:02:15 DEBUG "metric": { 13:02:15 DEBUG "pod": "ds-cts-0" 13:02:15 DEBUG }, 13:02:15 DEBUG "value": [ 13:02:15 DEBUG 13:02:15 DEBUG 13:02:15 DEBUG 13:02:15 DEBUG [...] 13:02:15 DEBUG 13:02:15 DEBUG 13:02:15 DEBUG 13:02:15 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:02:15 DEBUG }, 13:02:15 DEBUG "value": [ 13:02:15 DEBUG 1735909335, 13:02:15 DEBUG "0" 13:02:15 DEBUG ] 13:02:15 DEBUG } 13:02:15 DEBUG ] 13:02:15 DEBUG } 13:02:15 DEBUG } 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909336" 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG --- http response --- 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG { 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG "data": { 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG "result": [ 13:02:16 DEBUG { 13:02:16 DEBUG "metric": { 13:02:16 DEBUG "pod": "ds-cts-0" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG "pod": "ds-idrepo-0" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG "0" 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909336" 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909336" 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG --- http response --- 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:02:16 DEBUG --- http response --- 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:02:16 DEBUG { 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG { 13:02:16 DEBUG "data": { 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG "data": { 13:02:16 DEBUG "result": [ 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG { 13:02:16 DEBUG "result": [ 13:02:16 DEBUG "metric": { 13:02:16 DEBUG { 13:02:16 DEBUG "instance": "10.142.0.11:9100" 13:02:16 DEBUG "metric": { 13:02:16 DEBUG }, 13:02:16 DEBUG "pod": "ds-cts-0" 13:02:16 DEBUG "value": [ 13:02:16 DEBUG }, 13:02:16 DEBUG 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG "instance": "10.142.0.51:9100" 13:02:16 DEBUG 13:02:16 DEBUG }, 13:02:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:02:16 DEBUG "value": [ 13:02:16 DEBUG }, 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG "0.0011999999999981507" 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG ] 13:02:16 DEBUG "544.4093703272969" 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909336" 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG --- http response --- 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG { 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG "data": { 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG "result": [ 13:02:16 DEBUG { 13:02:16 DEBUG "metric": { 13:02:16 DEBUG "instance": "10.142.0.11:9100" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG "instance": "10.142.0.51:9100" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG "2377901.933333333" 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909336" 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG --- http response --- 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG { 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG "data": { 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG "result": [ 13:02:16 DEBUG { 13:02:16 DEBUG "metric": { 13:02:16 DEBUG "instance": "10.142.0.11:9100" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG "instance": "10.142.0.51:9100" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG "2395266.666666666" 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909336" 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG --- http response --- 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG { 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG "data": { 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG "result": [ 13:02:16 DEBUG { 13:02:16 DEBUG "metric": { 13:02:16 DEBUG "instance": "10.142.0.11:9100" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG "instance": "10.142.0.51:9100" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG "24439.466666666667" 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909336" 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG --- http response --- 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG { 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG "data": { 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG "result": [ 13:02:16 DEBUG { 13:02:16 DEBUG "metric": { 13:02:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG "0.0908651929498086" 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909336" 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909336" 13:02:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909336" 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG --- http response --- 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG { 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG "data": { 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG "result": [ 13:02:16 DEBUG { 13:02:16 DEBUG "metric": { 13:02:16 DEBUG "pod": "ds-idrepo-0" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG "758771712" 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG --- http response --- 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG { 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG "data": { 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG "result": [ 13:02:16 DEBUG { 13:02:16 DEBUG "metric": { 13:02:16 DEBUG "instance": "10.142.0.11:9100" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG "instance": "10.142.0.51:9100" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG "0.0010000000000009853" 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:16 INFO [http_cmd]: http status code OK 13:02:16 DEBUG --- status code --- 13:02:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:16 DEBUG --- http response --- 13:02:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:02:16 DEBUG ----- output ----- 13:02:16 DEBUG { 13:02:16 DEBUG "status": "success", 13:02:16 DEBUG "data": { 13:02:16 DEBUG "resultType": "vector", 13:02:16 DEBUG "result": [ 13:02:16 DEBUG { 13:02:16 DEBUG "metric": { 13:02:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG [...] 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG 13:02:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:02:16 DEBUG }, 13:02:16 DEBUG "value": [ 13:02:16 DEBUG 1735909336, 13:02:16 DEBUG "14995337216" 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG ] 13:02:16 DEBUG } 13:02:16 DEBUG } 13:02:23 INFO 13:02:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:02:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:02:23 INFO [loop_until]: OK (rc = 0) 13:02:23 DEBUG --- stdout --- 13:02:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 211m 1% 3840Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 180m 0% 19062Mi 67% 13:02:23 DEBUG --- stderr --- 13:02:23 DEBUG 13:02:27 INFO 13:02:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:02:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:02:27 INFO [loop_until]: OK (rc = 0) 13:02:27 DEBUG --- stdout --- 13:02:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 117m 13480Mi rcs-645c86fdd6-j4rsk 91m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:02:27 DEBUG --- stderr --- 13:02:27 DEBUG 13:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909394" 13:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909394" 13:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909394" 13:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909394" 13:03:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909394" 13:03:14 INFO [http_cmd]: http status code OK 13:03:14 ERROR [http_cmd]: ERROR 13:03:14 ERROR --- status code --- 13:03:14 INFO [http_cmd]: http status code OK 13:03:14 DEBUG --- status code --- 13:03:14 INFO [http_cmd]: http status code OK 13:03:14 INFO [http_cmd]: http status code OK 13:03:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:03:14 DEBUG --- status code --- 13:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:14 DEBUG --- status code --- 13:03:14 DEBUG --- status code --- 13:03:14 ERROR --- http response --- 13:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:14 DEBUG --- http response --- 13:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:03:14 DEBUG --- http response --- 13:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:14 DEBUG --- http response --- 13:03:14 DEBUG --- http response --- 13:03:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909395" 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:15 INFO [http_cmd]: http status code OK 13:03:15 DEBUG --- status code --- 13:03:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:15 DEBUG --- http response --- 13:03:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:03:15 DEBUG ----- output ----- 13:03:15 DEBUG { 13:03:15 DEBUG "status": "success", 13:03:15 DEBUG "data": { 13:03:15 DEBUG "resultType": "vector", 13:03:15 DEBUG "result": [ 13:03:15 DEBUG { 13:03:15 DEBUG "metric": { 13:03:15 DEBUG "pod": "ds-idrepo-0" 13:03:15 DEBUG }, 13:03:15 DEBUG "value": [ 13:03:15 DEBUG 13:03:15 DEBUG 13:03:15 DEBUG 13:03:15 DEBUG [...] 13:03:15 DEBUG 13:03:15 DEBUG 13:03:15 DEBUG 13:03:15 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:03:15 DEBUG }, 13:03:15 DEBUG "value": [ 13:03:15 DEBUG 1735909395, 13:03:15 DEBUG "0" 13:03:15 DEBUG ] 13:03:15 DEBUG } 13:03:15 DEBUG ] 13:03:15 DEBUG } 13:03:15 DEBUG } 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909396" 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG { 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "data": { 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG "result": [ 13:03:16 DEBUG { 13:03:16 DEBUG "metric": { 13:03:16 DEBUG "pod": "ds-cts-0" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG [...] 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "pod": "ds-idrepo-0" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG "0.027540247085895916" 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG } 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909396" 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909396" 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG { 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "data": { 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG "result": [ 13:03:16 DEBUG { 13:03:16 DEBUG "metric": { 13:03:16 DEBUG "instance": "10.142.0.11:9100" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG [...] 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "instance": "10.142.0.51:9100" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG "0.00043333333333066547" 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG } 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG { 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "data": { 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG "result": [ 13:03:16 DEBUG { 13:03:16 DEBUG "metric": { 13:03:16 DEBUG "pod": "ds-cts-0" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG [...] 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "pod": "ds-idrepo-0" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG "997.8075517661388" 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG } 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909396" 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG { 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "data": { 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG "result": [ 13:03:16 DEBUG { 13:03:16 DEBUG "metric": { 13:03:16 DEBUG "instance": "10.142.0.11:9100" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG [...] 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "instance": "10.142.0.51:9100" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG "1484589.8666666665" 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG } 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909396" 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG { 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "data": { 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG "result": [ 13:03:16 DEBUG { 13:03:16 DEBUG "metric": { 13:03:16 DEBUG "instance": "10.142.0.11:9100" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG [...] 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "instance": "10.142.0.51:9100" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG "1499385.133333333" 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG } 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909396" 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909396" 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG { 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "data": { 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG "result": [ 13:03:16 DEBUG { 13:03:16 DEBUG "metric": { 13:03:16 DEBUG "instance": "10.142.0.11:9100" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG [...] 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "instance": "10.142.0.51:9100" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG "23210.666666666664" 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG } 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG { 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "data": { 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG "result": [ 13:03:16 DEBUG { 13:03:16 DEBUG "metric": { 13:03:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG [...] 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG "0.038992334884248386" 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG } 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909396" 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909396" 13:03:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909396" 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG { 13:03:16 DEBUG { 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "data": { 13:03:16 INFO [http_cmd]: http status code OK 13:03:16 DEBUG "data": { 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG --- status code --- 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG "result": [ 13:03:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:16 DEBUG "result": [ 13:03:16 DEBUG { 13:03:16 DEBUG --- http response --- 13:03:16 DEBUG { 13:03:16 DEBUG "metric": { 13:03:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:03:16 DEBUG "metric": { 13:03:16 DEBUG "instance": "10.142.0.11:9100" 13:03:16 DEBUG ----- output ----- 13:03:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:03:16 DEBUG }, 13:03:16 DEBUG { 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG "status": "success", 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 13:03:16 DEBUG "data": { 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "resultType": "vector", 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "result": [ 13:03:16 DEBUG 13:03:16 DEBUG [...] 13:03:16 DEBUG { 13:03:16 DEBUG [...] 13:03:16 DEBUG 13:03:16 DEBUG "metric": { 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG "pod": "ds-idrepo-0" 13:03:16 DEBUG 13:03:16 DEBUG 13:03:16 DEBUG }, 13:03:16 DEBUG 13:03:16 DEBUG "instance": "10.142.0.51:9100" 13:03:16 DEBUG "value": [ 13:03:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:03:16 DEBUG }, 13:03:16 DEBUG 13:03:16 DEBUG }, 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG "0" 13:03:16 DEBUG [...] 13:03:16 DEBUG "14993686528" 13:03:16 DEBUG ] 13:03:16 DEBUG 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG 13:03:16 DEBUG } 13:03:16 DEBUG ] 13:03:16 DEBUG 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:03:16 DEBUG } 13:03:16 DEBUG } 13:03:16 DEBUG }, 13:03:16 DEBUG } 13:03:16 DEBUG "value": [ 13:03:16 DEBUG 1735909396, 13:03:16 DEBUG "758771712" 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG ] 13:03:16 DEBUG } 13:03:16 DEBUG } 13:03:23 INFO 13:03:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:03:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:03:23 INFO [loop_until]: OK (rc = 0) 13:03:23 DEBUG --- stdout --- 13:03:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 235m 1% 3846Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 194m 0% 19059Mi 67% 13:03:23 DEBUG --- stderr --- 13:03:23 DEBUG 13:03:27 INFO 13:03:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:03:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:03:27 INFO [loop_until]: OK (rc = 0) 13:03:27 DEBUG --- stdout --- 13:03:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 73m 13479Mi rcs-645c86fdd6-j4rsk 146m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:03:27 DEBUG --- stderr --- 13:03:27 DEBUG 13:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909454" 13:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909454" 13:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909454" 13:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909454" 13:04:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909454" 13:04:14 INFO [http_cmd]: http status code OK 13:04:14 DEBUG --- status code --- 13:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:14 DEBUG --- http response --- 13:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:14 INFO [http_cmd]: http status code OK 13:04:14 ERROR [http_cmd]: ERROR 13:04:14 DEBUG --- status code --- 13:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:14 INFO [http_cmd]: http status code OK 13:04:14 ERROR --- status code --- 13:04:14 DEBUG --- http response --- 13:04:14 DEBUG --- status code --- 13:04:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:14 ERROR --- http response --- 13:04:14 DEBUG --- http response --- 13:04:14 INFO [http_cmd]: http status code OK 13:04:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:14 DEBUG --- status code --- 13:04:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:04:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:14 DEBUG --- http response --- 13:04:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909455" 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:15 INFO [http_cmd]: http status code OK 13:04:15 DEBUG --- status code --- 13:04:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:15 DEBUG --- http response --- 13:04:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "pod": "ds-cts-0" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "pod": "ds-idrepo-0" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909455, 13:04:16 DEBUG "0" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909456" 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "pod": "ds-cts-0" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "0.00032394809741038814" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909456" 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909456" 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "instance": "10.142.0.11:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "instance": "10.142.0.51:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "0.0003999999999981204" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "pod": "ds-cts-0" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "0" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909456" 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "instance": "10.142.0.11:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "instance": "10.142.0.51:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "1218423.2666666668" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909456" 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "instance": "10.142.0.11:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "instance": "10.142.0.51:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "1235337.5000000005" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909456" 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909456" 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "instance": "10.142.0.11:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "instance": "10.142.0.51:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "18568.533333333333" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "0.04071734463295022" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909456" 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909456" 13:04:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909456" 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "pod": "ds-idrepo-0" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "758771712" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "instance": "10.142.0.11:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "instance": "10.142.0.51:9100" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "0.0003333333333330302" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 INFO [http_cmd]: http status code OK 13:04:16 DEBUG } 13:04:16 DEBUG --- status code --- 13:04:16 DEBUG } 13:04:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:16 DEBUG --- http response --- 13:04:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:04:16 DEBUG ----- output ----- 13:04:16 DEBUG { 13:04:16 DEBUG "status": "success", 13:04:16 DEBUG "data": { 13:04:16 DEBUG "resultType": "vector", 13:04:16 DEBUG "result": [ 13:04:16 DEBUG { 13:04:16 DEBUG "metric": { 13:04:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG [...] 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG 13:04:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:04:16 DEBUG }, 13:04:16 DEBUG "value": [ 13:04:16 DEBUG 1735909456, 13:04:16 DEBUG "14991224832" 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG ] 13:04:16 DEBUG } 13:04:16 DEBUG } 13:04:23 INFO 13:04:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:04:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:04:23 INFO [loop_until]: OK (rc = 0) 13:04:23 DEBUG --- stdout --- 13:04:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 184m 1% 3814Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 135m 0% 19056Mi 67% 13:04:23 DEBUG --- stderr --- 13:04:23 DEBUG 13:04:27 INFO 13:04:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:04:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:04:27 INFO [loop_until]: OK (rc = 0) 13:04:27 DEBUG --- stdout --- 13:04:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 97m 13479Mi rcs-645c86fdd6-j4rsk 56m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:04:27 DEBUG --- stderr --- 13:04:27 DEBUG 13:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909514" 13:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909514" 13:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909514" 13:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909514" 13:05:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909514" 13:05:14 ERROR [http_cmd]: ERROR 13:05:14 INFO [http_cmd]: http status code OK 13:05:14 ERROR --- status code --- 13:05:14 INFO [http_cmd]: http status code OK 13:05:14 DEBUG --- status code --- 13:05:14 INFO [http_cmd]: http status code OK 13:05:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:05:14 DEBUG --- status code --- 13:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:14 DEBUG --- status code --- 13:05:14 ERROR --- http response --- 13:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:14 DEBUG --- http response --- 13:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:14 INFO [http_cmd]: http status code OK 13:05:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:05:14 DEBUG --- http response --- 13:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:14 DEBUG --- http response --- 13:05:14 DEBUG --- status code --- 13:05:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:14 DEBUG --- http response --- 13:05:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG --- http response --- 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG --- status code --- 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 DEBUG --- http response --- 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 DEBUG --- status code --- 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909515" 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 INFO [http_cmd]: http status code OK 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG --- status code --- 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:15 DEBUG --- http response --- 13:05:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909515" 13:05:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909515" 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909516" 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909516" 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909516" 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG --- http response --- 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG { 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG "data": { 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "result": [ 13:05:16 DEBUG { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "pod": "ds-idrepo-0" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG [...] 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG "pod": "ds-cts-0" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG "0" 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909516" 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG { 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG "data": { 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "result": [ 13:05:16 DEBUG { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "pod": "ds-idrepo-0" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG [...] 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG "pod": "ds-cts-0" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG "0.0019290438349888574" 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909516" 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909516" 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG { 13:05:16 DEBUG { 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG "data": { 13:05:16 DEBUG "data": { 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "result": [ 13:05:16 DEBUG "result": [ 13:05:16 DEBUG { 13:05:16 DEBUG { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "pod": "ds-idrepo-0" 13:05:16 DEBUG "instance": "10.142.0.11:9100" 13:05:16 DEBUG }, 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG [...] 13:05:16 DEBUG [...] 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG "pod": "ds-cts-0" 13:05:16 DEBUG "instance": "10.142.0.51:9100" 13:05:16 DEBUG }, 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG "0" 13:05:16 DEBUG "0.00026666666666793997" 13:05:16 DEBUG ] 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 DEBUG ] 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909516" 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG { 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG "data": { 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "result": [ 13:05:16 DEBUG { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "instance": "10.142.0.11:9100" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG [...] 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG "instance": "10.142.0.51:9100" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG "1144158.7333333327" 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909516" 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG { 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG "data": { 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "result": [ 13:05:16 DEBUG { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "instance": "10.142.0.11:9100" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG [...] 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG "instance": "10.142.0.51:9100" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG "1161141.1000000003" 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909516" 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909516" 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG { 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG "data": { 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "result": [ 13:05:16 DEBUG { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "instance": "10.142.0.11:9100" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG [...] 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG "instance": "10.142.0.51:9100" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG "8601.6" 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG { 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG "data": { 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "result": [ 13:05:16 DEBUG { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG [...] 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG "0.041929912744495215" 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909516" 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909516" 13:05:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909516" 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 INFO [http_cmd]: http status code OK 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:05:16 DEBUG --- status code --- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG { 13:05:16 DEBUG --- http response --- 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:05:16 DEBUG "data": { 13:05:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG ----- output ----- 13:05:16 DEBUG { 13:05:16 DEBUG "result": [ 13:05:16 DEBUG { 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG { 13:05:16 DEBUG "status": "success", 13:05:16 DEBUG "data": { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "data": { 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "instance": "10.142.0.11:9100" 13:05:16 DEBUG "resultType": "vector", 13:05:16 DEBUG "result": [ 13:05:16 DEBUG }, 13:05:16 DEBUG "result": [ 13:05:16 DEBUG { 13:05:16 DEBUG "value": [ 13:05:16 DEBUG { 13:05:16 DEBUG "metric": { 13:05:16 DEBUG 13:05:16 DEBUG "metric": { 13:05:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:05:16 DEBUG 13:05:16 DEBUG "pod": "ds-idrepo-0" 13:05:16 DEBUG }, 13:05:16 DEBUG 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG [...] 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG [...] 13:05:16 DEBUG "instance": "10.142.0.51:9100" 13:05:16 DEBUG [...] 13:05:16 DEBUG 13:05:16 DEBUG }, 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 13:05:16 DEBUG 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG 13:05:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:05:16 DEBUG "0" 13:05:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:05:16 DEBUG }, 13:05:16 DEBUG ] 13:05:16 DEBUG }, 13:05:16 DEBUG "value": [ 13:05:16 DEBUG } 13:05:16 DEBUG "value": [ 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG ] 13:05:16 DEBUG 1735909516, 13:05:16 DEBUG "14990852096" 13:05:16 DEBUG } 13:05:16 DEBUG "758779904" 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 DEBUG ] 13:05:16 DEBUG ] 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:16 DEBUG } 13:05:23 INFO 13:05:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:05:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:05:23 INFO [loop_until]: OK (rc = 0) 13:05:23 DEBUG --- stdout --- 13:05:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 168m 1% 3836Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 130m 0% 19061Mi 67% 13:05:23 DEBUG --- stderr --- 13:05:23 DEBUG 13:05:27 INFO 13:05:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:05:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:05:27 INFO [loop_until]: OK (rc = 0) 13:05:27 DEBUG --- stdout --- 13:05:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 57m 13479Mi rcs-645c86fdd6-j4rsk 29m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:05:27 DEBUG --- stderr --- 13:05:27 DEBUG 13:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909574" 13:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909574" 13:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909574" 13:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909574" 13:06:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909574" 13:06:14 INFO [http_cmd]: http status code OK 13:06:14 DEBUG --- status code --- 13:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:14 ERROR [http_cmd]: ERROR 13:06:14 INFO [http_cmd]: http status code OK 13:06:14 DEBUG --- http response --- 13:06:14 INFO [http_cmd]: http status code OK 13:06:14 ERROR --- status code --- 13:06:14 DEBUG --- status code --- 13:06:14 DEBUG --- status code --- 13:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:14 INFO [http_cmd]: http status code OK 13:06:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:14 DEBUG --- http response --- 13:06:14 ERROR --- http response --- 13:06:14 DEBUG --- http response --- 13:06:14 DEBUG --- status code --- 13:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:06:14 DEBUG --- http response --- 13:06:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909575" 13:06:15 DEBUG --- status code --- 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- http response --- 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909575" 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:15 INFO [http_cmd]: http status code OK 13:06:15 DEBUG --- status code --- 13:06:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:15 DEBUG --- http response --- 13:06:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG { 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG "data": { 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG "result": [ 13:06:16 DEBUG { 13:06:16 DEBUG "metric": { 13:06:16 DEBUG "pod": "ds-cts-0" 13:06:16 DEBUG }, 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG [...] 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:06:16 DEBUG }, 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG "0" 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG } 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG { 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG "data": { 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG "result": [ 13:06:16 DEBUG { 13:06:16 DEBUG "metric": { 13:06:16 DEBUG "pod": "ds-cts-0" 13:06:16 DEBUG }, 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG [...] 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:06:16 DEBUG }, 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG "0" 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG } 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909576" 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG { 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG "data": { 13:06:16 DEBUG { 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG "result": [ 13:06:16 DEBUG "data": { 13:06:16 DEBUG { 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG "metric": { 13:06:16 DEBUG "result": [ 13:06:16 DEBUG "pod": "ds-cts-0" 13:06:16 DEBUG { 13:06:16 DEBUG }, 13:06:16 DEBUG "metric": { 13:06:16 DEBUG "value": [ 13:06:16 DEBUG "instance": "10.142.0.11:9100" 13:06:16 DEBUG 13:06:16 DEBUG }, 13:06:16 DEBUG 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG [...] 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG [...] 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:06:16 DEBUG 13:06:16 DEBUG }, 13:06:16 DEBUG 13:06:16 DEBUG "value": [ 13:06:16 DEBUG "instance": "10.142.0.51:9100" 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG }, 13:06:16 DEBUG "308.216534822398" 13:06:16 DEBUG "value": [ 13:06:16 DEBUG ] 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG } 13:06:16 DEBUG "0.00043333333333066547" 13:06:16 DEBUG ] 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG } 13:06:16 DEBUG } 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG } 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909576" 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG { 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG "data": { 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG "result": [ 13:06:16 DEBUG { 13:06:16 DEBUG "metric": { 13:06:16 DEBUG "instance": "10.142.0.11:9100" 13:06:16 DEBUG }, 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG [...] 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG "instance": "10.142.0.51:9100" 13:06:16 DEBUG }, 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG "260247.76666666675" 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG } 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909576" 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG { 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG "data": { 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG "result": [ 13:06:16 DEBUG { 13:06:16 DEBUG "metric": { 13:06:16 DEBUG "instance": "10.142.0.11:9100" 13:06:16 DEBUG }, 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG [...] 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG "instance": "10.142.0.51:9100" 13:06:16 DEBUG }, 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG "277667.36666666664" 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG } 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909576" 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG { 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG "data": { 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG "result": [ 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:06:16 DEBUG { 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG "metric": { 13:06:16 DEBUG { 13:06:16 DEBUG "instance": "10.142.0.11:9100" 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG }, 13:06:16 DEBUG "data": { 13:06:16 DEBUG "value": [ 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG 13:06:16 DEBUG "result": [ 13:06:16 DEBUG 13:06:16 DEBUG { 13:06:16 DEBUG 13:06:16 DEBUG "metric": { 13:06:16 DEBUG [...] 13:06:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:06:16 DEBUG 13:06:16 DEBUG }, 13:06:16 DEBUG 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG "instance": "10.142.0.51:9100" 13:06:16 DEBUG 13:06:16 DEBUG }, 13:06:16 DEBUG 13:06:16 DEBUG "value": [ 13:06:16 DEBUG [...] 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG 13:06:16 DEBUG "25531.73333333333" 13:06:16 DEBUG 13:06:16 DEBUG ] 13:06:16 DEBUG 13:06:16 DEBUG } 13:06:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:06:16 DEBUG ] 13:06:16 DEBUG }, 13:06:16 DEBUG } 13:06:16 DEBUG "value": [ 13:06:16 DEBUG } 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG "0.10452831329744337" 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG } 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909576" 13:06:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909576" 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG { 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG "status": "success", 13:06:16 INFO [http_cmd]: http status code OK 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG "data": { 13:06:16 DEBUG --- status code --- 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG "result": [ 13:06:16 DEBUG --- http response --- 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:06:16 DEBUG { 13:06:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG "metric": { 13:06:16 DEBUG "pod": "ds-idrepo-0" 13:06:16 DEBUG { 13:06:16 DEBUG ----- output ----- 13:06:16 DEBUG }, 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG { 13:06:16 DEBUG "value": [ 13:06:16 DEBUG "data": { 13:06:16 DEBUG "status": "success", 13:06:16 DEBUG 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG "data": { 13:06:16 DEBUG 13:06:16 DEBUG "result": [ 13:06:16 DEBUG "resultType": "vector", 13:06:16 DEBUG 13:06:16 DEBUG { 13:06:16 DEBUG "result": [ 13:06:16 DEBUG [...] 13:06:16 DEBUG "metric": { 13:06:16 DEBUG { 13:06:16 DEBUG 13:06:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:06:16 DEBUG }, 13:06:16 DEBUG 13:06:16 DEBUG "metric": { 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 13:06:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:06:16 DEBUG 13:06:16 DEBUG "instance": "10.142.0.11:9100" 13:06:16 DEBUG }, 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG "value": [ 13:06:16 DEBUG }, 13:06:16 DEBUG [...] 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG "value": [ 13:06:16 DEBUG 13:06:16 DEBUG "758779904" 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG 13:06:16 DEBUG 13:06:16 DEBUG ] 13:06:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:06:16 DEBUG }, 13:06:16 DEBUG } 13:06:16 DEBUG 13:06:16 DEBUG "value": [ 13:06:16 DEBUG } 13:06:16 DEBUG [...] 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG 13:06:16 DEBUG "14995525632" 13:06:16 DEBUG 13:06:16 DEBUG ] 13:06:16 DEBUG 13:06:16 DEBUG } 13:06:16 DEBUG "instance": "10.142.0.51:9100" 13:06:16 DEBUG ] 13:06:16 DEBUG }, 13:06:16 DEBUG } 13:06:16 DEBUG "value": [ 13:06:16 DEBUG } 13:06:16 DEBUG 1735909576, 13:06:16 DEBUG "0.0006666666666679551" 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG ] 13:06:16 DEBUG } 13:06:16 DEBUG } 13:06:23 INFO 13:06:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:06:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:06:23 INFO [loop_until]: OK (rc = 0) 13:06:23 DEBUG --- stdout --- 13:06:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 160m 1% 3824Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 139m 0% 19059Mi 67% 13:06:23 DEBUG --- stderr --- 13:06:23 DEBUG 13:06:27 INFO 13:06:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:06:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:06:27 INFO [loop_until]: OK (rc = 0) 13:06:27 DEBUG --- stdout --- 13:06:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 75m 13480Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:06:27 DEBUG --- stderr --- 13:06:27 DEBUG 13:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909634" 13:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909634" 13:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909634" 13:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909634" 13:07:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909634" 13:07:14 INFO [http_cmd]: http status code OK 13:07:14 DEBUG --- status code --- 13:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:14 DEBUG --- http response --- 13:07:14 ERROR [http_cmd]: ERROR 13:07:14 ERROR --- status code --- 13:07:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:07:14 ERROR --- http response --- 13:07:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:07:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:14 INFO [http_cmd]: http status code OK 13:07:14 DEBUG --- status code --- 13:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:14 DEBUG --- http response --- 13:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:14 INFO [http_cmd]: http status code OK 13:07:14 DEBUG --- status code --- 13:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:14 DEBUG --- http response --- 13:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:14 INFO [http_cmd]: http status code OK 13:07:14 DEBUG --- status code --- 13:07:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:14 DEBUG --- http response --- 13:07:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909635" 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG --- status code --- 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909635" 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:15 INFO [http_cmd]: http status code OK 13:07:15 DEBUG --- status code --- 13:07:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:15 DEBUG --- http response --- 13:07:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG { 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG "data": { 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG "result": [ 13:07:16 DEBUG { 13:07:16 DEBUG "metric": { 13:07:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG [...] 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG "pod": "ds-cts-0" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG "0" 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG { 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG "data": { 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG "result": [ 13:07:16 DEBUG { 13:07:16 DEBUG "metric": { 13:07:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG [...] 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG "pod": "ds-cts-0" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG "0.0027333562231722313" 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909636" 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG { 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG "data": { 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG "result": [ 13:07:16 DEBUG { 13:07:16 DEBUG "metric": { 13:07:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG [...] 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG "pod": "ds-cts-0" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG "562.5407725321888" 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG { 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG "data": { 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG "result": [ 13:07:16 DEBUG { 13:07:16 DEBUG "metric": { 13:07:16 DEBUG "instance": "10.142.0.11:9100" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG [...] 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG "instance": "10.142.0.51:9100" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG "0.0011999999999981507" 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909636" 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG { 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG "data": { 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG "result": [ 13:07:16 DEBUG { 13:07:16 DEBUG "metric": { 13:07:16 DEBUG "instance": "10.142.0.11:9100" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG [...] 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG "instance": "10.142.0.51:9100" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG "72779.73333333332" 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909636" 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG { 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG "data": { 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG "result": [ 13:07:16 DEBUG { 13:07:16 DEBUG "metric": { 13:07:16 DEBUG "instance": "10.142.0.11:9100" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG [...] 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG "instance": "10.142.0.51:9100" 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG "89542.50000000001" 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909636" 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG { 13:07:16 DEBUG { 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG "data": { 13:07:16 DEBUG "data": { 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG "result": [ 13:07:16 DEBUG "result": [ 13:07:16 DEBUG { 13:07:16 DEBUG { 13:07:16 DEBUG "metric": { 13:07:16 DEBUG "metric": { 13:07:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:07:16 DEBUG "instance": "10.142.0.11:9100" 13:07:16 DEBUG }, 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG [...] 13:07:16 DEBUG [...] 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:07:16 DEBUG "instance": "10.142.0.51:9100" 13:07:16 DEBUG }, 13:07:16 DEBUG }, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG "0.6508398506750666" 13:07:16 DEBUG "24166.4" 13:07:16 DEBUG ] 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909636" 13:07:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909636" 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- http response --- 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:07:16 DEBUG --- status code --- 13:07:16 INFO [http_cmd]: http status code OK 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG --- status code --- 13:07:16 DEBUG { 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:07:16 DEBUG --- http response --- 13:07:16 DEBUG "data": { 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG { 13:07:16 DEBUG ----- output ----- 13:07:16 DEBUG "result": [ 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG { 13:07:16 DEBUG { 13:07:16 DEBUG "data": { 13:07:16 DEBUG "status": "success", 13:07:16 DEBUG "metric": { 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG "data": { 13:07:16 DEBUG "pod": "ds-idrepo-0" 13:07:16 DEBUG "result": [ 13:07:16 DEBUG "resultType": "vector", 13:07:16 DEBUG }, 13:07:16 DEBUG { 13:07:16 DEBUG "result": [ 13:07:16 DEBUG "value": [ 13:07:16 DEBUG "metric": { 13:07:16 DEBUG { 13:07:16 DEBUG 13:07:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:07:16 DEBUG "metric": { 13:07:16 DEBUG 13:07:16 DEBUG }, 13:07:16 DEBUG "instance": "10.142.0.11:9100" 13:07:16 DEBUG 13:07:16 DEBUG "value": [ 13:07:16 DEBUG }, 13:07:16 DEBUG [...] 13:07:16 DEBUG 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG [...] 13:07:16 DEBUG 13:07:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:07:16 DEBUG 13:07:16 DEBUG [...] 13:07:16 DEBUG }, 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG "value": [ 13:07:16 DEBUG 13:07:16 DEBUG 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:07:16 DEBUG 13:07:16 DEBUG "758771712" 13:07:16 DEBUG }, 13:07:16 DEBUG "instance": "10.142.0.51:9100" 13:07:16 DEBUG ] 13:07:16 DEBUG "value": [ 13:07:16 DEBUG }, 13:07:16 DEBUG } 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG "value": [ 13:07:16 DEBUG ] 13:07:16 DEBUG "14992674816" 13:07:16 DEBUG 1735909636, 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG "0.0006666666666660604" 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 DEBUG ] 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:16 DEBUG } 13:07:23 INFO 13:07:23 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:07:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:07:24 INFO [loop_until]: OK (rc = 0) 13:07:24 DEBUG --- stdout --- 13:07:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 110m 0% 3835Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 83m 0% 19054Mi 66% 13:07:24 DEBUG --- stderr --- 13:07:24 DEBUG 13:07:27 INFO 13:07:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:07:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:07:27 INFO [loop_until]: OK (rc = 0) 13:07:27 DEBUG --- stdout --- 13:07:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 6m 13479Mi rcs-645c86fdd6-j4rsk 19m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:07:27 DEBUG --- stderr --- 13:07:27 DEBUG 13:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909694" 13:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909694" 13:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909694" 13:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909694" 13:08:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909694" 13:08:14 INFO [http_cmd]: http status code OK 13:08:14 INFO [http_cmd]: http status code OK 13:08:14 ERROR [http_cmd]: ERROR 13:08:14 DEBUG --- status code --- 13:08:14 INFO [http_cmd]: http status code OK 13:08:14 DEBUG --- status code --- 13:08:14 INFO [http_cmd]: http status code OK 13:08:14 ERROR --- status code --- 13:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:14 DEBUG --- status code --- 13:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:14 DEBUG --- status code --- 13:08:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:08:14 DEBUG --- http response --- 13:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:14 DEBUG --- http response --- 13:08:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:14 ERROR --- http response --- 13:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:14 DEBUG --- http response --- 13:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:14 DEBUG --- http response --- 13:08:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909695" 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 INFO [http_cmd]: http status code OK 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG --- status code --- 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG --- http response --- 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735909696, "0" ] } ] } } 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:08:16 DEBUG ----- output ----- 13:08:16 DEBUG { 13:08:16 DEBUG "status": "success", 13:08:16 DEBUG "data": { 13:08:16 DEBUG "resultType": "vector", 13:08:16 DEBUG "result": [ 13:08:16 DEBUG { 13:08:16 DEBUG "metric": { 13:08:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG [...] 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG "pod": "ds-cts-0" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 1735909696, 13:08:16 DEBUG "0.0012785391681004862" 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG } 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909696" 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735909696, "438.7296470924418" ] } ] } } 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:08:16 DEBUG ----- output ----- 13:08:16 DEBUG { 13:08:16 DEBUG "status": "success", 13:08:16 DEBUG "data": { 13:08:16 DEBUG "resultType": "vector", 13:08:16 DEBUG "result": [ 13:08:16 DEBUG { 13:08:16 DEBUG "metric": { 13:08:16 DEBUG "instance": "10.142.0.11:9100" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG [...] 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG "instance": "10.142.0.51:9100" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 1735909696, 13:08:16 DEBUG "0.00043333333333824464" 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG } 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909696" 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:08:16 DEBUG ----- output ----- 13:08:16 DEBUG { 13:08:16 DEBUG "status": "success", 13:08:16 DEBUG "data": { 13:08:16 DEBUG "resultType": "vector", 13:08:16 DEBUG "result": [ 13:08:16 DEBUG { 13:08:16 DEBUG "metric": { 13:08:16 DEBUG "instance": "10.142.0.11:9100" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG [...] 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG "instance": "10.142.0.51:9100" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 1735909696, 13:08:16 DEBUG "77356.23333333334" 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG } 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909696" 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:08:16 DEBUG ----- output ----- 13:08:16 DEBUG { 13:08:16 DEBUG "status": "success", 13:08:16 DEBUG "data": { 13:08:16 DEBUG "resultType": "vector", 13:08:16 DEBUG "result": [ 13:08:16 DEBUG { 13:08:16 DEBUG "metric": { 13:08:16 DEBUG "instance": "10.142.0.11:9100" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG [...] 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG "instance": "10.142.0.51:9100" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 1735909696, 13:08:16 DEBUG "92116.46666666667" 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG } 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909696" 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:08:16 DEBUG ----- output ----- 13:08:16 DEBUG { 13:08:16 DEBUG "status": "success", 13:08:16 DEBUG "data": { 13:08:16 DEBUG "resultType": "vector", 13:08:16 DEBUG "result": [ 13:08:16 DEBUG { 13:08:16 DEBUG "metric": { 13:08:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG [...] 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 1735909696, 13:08:16 DEBUG "0.11476928589737168" 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG } 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:08:16 DEBUG ----- output ----- 13:08:16 DEBUG { 13:08:16 DEBUG "status": "success", 13:08:16 DEBUG "data": { 13:08:16 DEBUG "resultType": "vector", 13:08:16 DEBUG "result": [ 13:08:16 DEBUG { 13:08:16 DEBUG "metric": { 13:08:16 DEBUG "instance": "10.142.0.11:9100" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG [...] 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG "instance": "10.142.0.51:9100" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 1735909696, 13:08:16 DEBUG "21708.8" 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG } 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909696" 13:08:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909696" 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:08:16 DEBUG ----- output ----- 13:08:16 DEBUG { 13:08:16 DEBUG "status": "success", 13:08:16 DEBUG "data": { 13:08:16 DEBUG "resultType": "vector", 13:08:16 DEBUG "result": [ 13:08:16 DEBUG { 13:08:16 DEBUG "metric": { 13:08:16 DEBUG "pod": "ds-idrepo-0" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG [...] 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:08:16 DEBUG }, 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 1735909696, 13:08:16 DEBUG "758771712" 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG } 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:08:16 INFO [http_cmd]: http status code OK 13:08:16 DEBUG ----- output ----- 13:08:16 DEBUG --- status code --- 13:08:16 DEBUG { 13:08:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:16 DEBUG "status": "success", 13:08:16 DEBUG --- http response --- 13:08:16 DEBUG "data": { 13:08:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:08:16 DEBUG "resultType": "vector", 13:08:16 DEBUG ----- output ----- 13:08:16 DEBUG "result": [ 13:08:16 DEBUG { 13:08:16 DEBUG { 13:08:16 DEBUG "status": "success", 13:08:16 DEBUG "metric": { 13:08:16 DEBUG "data": { 13:08:16 DEBUG "instance": "10.142.0.11:9100" 13:08:16 DEBUG "resultType": "vector", 13:08:16 DEBUG }, 13:08:16 DEBUG "result": [ 13:08:16 DEBUG "value": [ 13:08:16 DEBUG { 13:08:16 DEBUG 13:08:16 DEBUG "metric": { 13:08:16 DEBUG 13:08:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:08:16 DEBUG 13:08:16 DEBUG }, 13:08:16 DEBUG [...] 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG 13:08:16 DEBUG "instance": "10.142.0.51:9100" 13:08:16 DEBUG [...] 13:08:16 DEBUG }, 13:08:16 DEBUG 13:08:16 DEBUG "value": [ 13:08:16 DEBUG 13:08:16 DEBUG 1735909696, 13:08:16 DEBUG 13:08:16 DEBUG "0.00033333333333350387" 13:08:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:08:16 DEBUG ] 13:08:16 DEBUG }, 13:08:16 DEBUG } 13:08:16 DEBUG "value": [ 13:08:16 DEBUG ] 13:08:16 DEBUG 1735909696, 13:08:16 DEBUG } 13:08:16 DEBUG "14988439552" 13:08:16 DEBUG } 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG ] 13:08:16 DEBUG } 13:08:16 DEBUG } 13:08:24 INFO 13:08:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:08:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:08:24 INFO [loop_until]: OK (rc = 0) 13:08:24 DEBUG --- stdout --- 13:08:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 101m 0% 3818Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 74m 0% 19060Mi 67% 13:08:24 DEBUG --- stderr --- 13:08:24 DEBUG 13:08:27 INFO 13:08:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:08:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:08:27 INFO [loop_until]: OK (rc = 0) 13:08:27 DEBUG --- stdout --- 13:08:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13480Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:08:27 DEBUG --- stderr --- 13:08:27 DEBUG 13:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909754" 13:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909754" 13:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909754" 13:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909754" 13:09:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909754" 13:09:14 INFO [http_cmd]: http status code OK 13:09:14 DEBUG --- status code --- 13:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:14 DEBUG --- http response --- 13:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:14 INFO [http_cmd]: http status code OK 13:09:14 DEBUG --- status code --- 13:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:14 DEBUG --- http response --- 13:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:14 INFO [http_cmd]: http status code OK 13:09:14 DEBUG --- status code --- 13:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:14 DEBUG --- http response --- 13:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:14 ERROR [http_cmd]: ERROR 13:09:14 ERROR --- status code --- 13:09:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:09:14 ERROR --- http response --- 13:09:14 INFO [http_cmd]: http status code OK 13:09:14 DEBUG --- status code --- 13:09:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:14 DEBUG --- http response --- 13:09:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:09:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909755" 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 INFO [http_cmd]: http status code OK 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG --- status code --- 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG --- http response --- 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:16 DEBUG --- http response --- 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG { 13:09:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "data": { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG { 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "pod": "ds-cts-0" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG "0" 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG { 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "data": { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG { 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "pod": "ds-cts-0" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG "0" 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909756" 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG { 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "data": { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG { 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "pod": "ds-cts-0" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG "0" 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG { 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "data": { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG { 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "instance": "10.142.0.11:9100" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "instance": "10.142.0.51:9100" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG "0.000733333333336835" 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909756" 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG { 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "data": { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG { 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "instance": "10.142.0.11:9100" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "instance": "10.142.0.51:9100" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG "327229.4666666666" 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909756" 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG { 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "data": { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG { 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "instance": "10.142.0.11:9100" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "instance": "10.142.0.51:9100" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG "344144.56666666665" 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909756" 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG { 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG "data": { 13:09:16 DEBUG { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG "data": { 13:09:16 DEBUG { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "result": [ 13:09:16 DEBUG "instance": "10.142.0.11:9100" 13:09:16 DEBUG { 13:09:16 DEBUG }, 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "value": [ 13:09:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:09:16 DEBUG 13:09:16 DEBUG }, 13:09:16 DEBUG 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "instance": "10.142.0.51:9100" 13:09:16 DEBUG 13:09:16 DEBUG }, 13:09:16 DEBUG 13:09:16 DEBUG "value": [ 13:09:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG }, 13:09:16 DEBUG "20753.066666666666" 13:09:16 DEBUG "value": [ 13:09:16 DEBUG ] 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG } 13:09:16 DEBUG "0.1689680667593638" 13:09:16 DEBUG ] 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909756" 13:09:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909756" 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG { 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "data": { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG { 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "pod": "ds-idrepo-0" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG "758771712" 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG { 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "data": { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG { 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "instance": "10.142.0.11:9100" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "instance": "10.142.0.51:9100" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG "0.00033333333333397756" 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:16 INFO [http_cmd]: http status code OK 13:09:16 DEBUG --- status code --- 13:09:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:16 DEBUG --- http response --- 13:09:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:09:16 DEBUG ----- output ----- 13:09:16 DEBUG { 13:09:16 DEBUG "status": "success", 13:09:16 DEBUG "data": { 13:09:16 DEBUG "resultType": "vector", 13:09:16 DEBUG "result": [ 13:09:16 DEBUG { 13:09:16 DEBUG "metric": { 13:09:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG [...] 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG 13:09:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:09:16 DEBUG }, 13:09:16 DEBUG "value": [ 13:09:16 DEBUG 1735909756, 13:09:16 DEBUG "14991986688" 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG ] 13:09:16 DEBUG } 13:09:16 DEBUG } 13:09:24 INFO 13:09:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:09:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:09:24 INFO [loop_until]: OK (rc = 0) 13:09:24 DEBUG --- stdout --- 13:09:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 106m 0% 3844Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 76m 0% 19059Mi 67% 13:09:24 DEBUG --- stderr --- 13:09:24 DEBUG 13:09:27 INFO 13:09:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:09:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:09:27 INFO [loop_until]: OK (rc = 0) 13:09:27 DEBUG --- stdout --- 13:09:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 71m 13479Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:09:27 DEBUG --- stderr --- 13:09:27 DEBUG 13:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909814" 13:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909814" 13:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909814" 13:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909814" 13:10:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909814" 13:10:14 INFO [http_cmd]: http status code OK 13:10:14 DEBUG --- status code --- 13:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:14 DEBUG --- http response --- 13:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:14 ERROR [http_cmd]: ERROR 13:10:14 ERROR --- status code --- 13:10:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:10:14 ERROR --- http response --- 13:10:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:10:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:10:14 INFO [http_cmd]: http status code OK 13:10:14 DEBUG --- status code --- 13:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:14 DEBUG --- http response --- 13:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:14 INFO [http_cmd]: http status code OK 13:10:14 DEBUG --- status code --- 13:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:14 DEBUG --- http response --- 13:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:14 INFO [http_cmd]: http status code OK 13:10:14 DEBUG --- status code --- 13:10:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:14 DEBUG --- http response --- 13:10:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909815" 13:10:15 DEBUG --- http response --- 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909815" 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909815" 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 INFO [http_cmd]: http status code OK 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG --- status code --- 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG --- http response --- 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG { 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "data": { 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "result": [ 13:10:16 DEBUG { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG "pod": "ds-cts-0" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "0" 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG { 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "data": { 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "result": [ 13:10:16 DEBUG { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG "pod": "ds-cts-0" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "0.001317843899247391" 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909816" 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG { 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "data": { 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "result": [ 13:10:16 DEBUG { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG "pod": "ds-cts-0" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "453.261282336199" 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG { 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "data": { 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "result": [ 13:10:16 DEBUG { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "instance": "10.142.0.11:9100" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG "instance": "10.142.0.51:9100" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "0.00046666666666321055" 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909816" 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG { 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "data": { 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "result": [ 13:10:16 DEBUG { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "instance": "10.142.0.11:9100" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG "instance": "10.142.0.51:9100" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "1222778.233333333" 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909816" 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG { 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "data": { 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "result": [ 13:10:16 DEBUG { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "instance": "10.142.0.11:9100" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG "instance": "10.142.0.51:9100" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "1240178.3333333335" 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909816" 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG { 13:10:16 DEBUG { 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "data": { 13:10:16 DEBUG "data": { 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "result": [ 13:10:16 DEBUG "result": [ 13:10:16 DEBUG { 13:10:16 DEBUG { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "instance": "10.142.0.11:9100" 13:10:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:10:16 DEBUG }, 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG "instance": "10.142.0.51:9100" 13:10:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:10:16 DEBUG }, 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "23620.266666666663" 13:10:16 DEBUG "0.1831890183191377" 13:10:16 DEBUG ] 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 DEBUG ] 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909816" 13:10:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909816" 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG { 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "data": { 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "result": [ 13:10:16 DEBUG { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "pod": "ds-idrepo-0" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "758771712" 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 INFO [http_cmd]: http status code OK 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG --- status code --- 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG --- http response --- 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG { 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG "data": { 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG "result": [ 13:10:16 DEBUG { 13:10:16 DEBUG "metric": { 13:10:16 DEBUG "instance": "10.142.0.11:9100" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:10:16 DEBUG "instance": "10.142.0.51:9100" 13:10:16 DEBUG ----- output ----- 13:10:16 DEBUG }, 13:10:16 DEBUG { 13:10:16 DEBUG "value": [ 13:10:16 DEBUG "status": "success", 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "data": { 13:10:16 DEBUG "0.00033333333333397756" 13:10:16 DEBUG "resultType": "vector", 13:10:16 DEBUG ] 13:10:16 DEBUG "result": [ 13:10:16 DEBUG } 13:10:16 DEBUG { 13:10:16 DEBUG ] 13:10:16 DEBUG "metric": { 13:10:16 DEBUG } 13:10:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:10:16 DEBUG } 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG [...] 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG 13:10:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:10:16 DEBUG }, 13:10:16 DEBUG "value": [ 13:10:16 DEBUG 1735909816, 13:10:16 DEBUG "14992224256" 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG ] 13:10:16 DEBUG } 13:10:16 DEBUG } 13:10:24 INFO 13:10:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:10:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:10:24 INFO [loop_until]: OK (rc = 0) 13:10:24 DEBUG --- stdout --- 13:10:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 100m 0% 3839Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 82m 0% 19057Mi 67% 13:10:24 DEBUG --- stderr --- 13:10:24 DEBUG 13:10:27 INFO 13:10:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:10:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:10:27 INFO [loop_until]: OK (rc = 0) 13:10:27 DEBUG --- stdout --- 13:10:27 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 59m 13480Mi rcs-645c86fdd6-j4rsk 48m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:10:27 DEBUG --- stderr --- 13:10:27 DEBUG 13:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909874" 13:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909874" 13:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909874" 13:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909874" 13:11:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909874" 13:11:14 INFO [http_cmd]: http status code OK 13:11:14 DEBUG --- status code --- 13:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:14 DEBUG --- http response --- 13:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:14 INFO [http_cmd]: http status code OK 13:11:14 DEBUG --- status code --- 13:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:14 DEBUG --- http response --- 13:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:14 ERROR [http_cmd]: ERROR 13:11:14 ERROR --- status code --- 13:11:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:11:14 ERROR --- http response --- 13:11:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:11:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:11:14 INFO [http_cmd]: http status code OK 13:11:14 DEBUG --- status code --- 13:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:14 DEBUG --- http response --- 13:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:14 INFO [http_cmd]: http status code OK 13:11:14 DEBUG --- status code --- 13:11:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:14 DEBUG --- http response --- 13:11:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909875" 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:15 INFO [http_cmd]: http status code OK 13:11:15 DEBUG --- status code --- 13:11:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:15 DEBUG --- http response --- 13:11:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG --- http response --- 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735909876, "0" ] } ] } } 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735909876, "0.0007737977872718231" ] } ] } } 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909876" 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735909876, "299.1200204476576" ] } ] } } 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:11:16 DEBUG ----- output ----- 13:11:16 DEBUG { 13:11:16 DEBUG "status": "success", 13:11:16 DEBUG "data": { 13:11:16 DEBUG "resultType": "vector", 13:11:16 DEBUG "result": [ 13:11:16 DEBUG { 13:11:16 DEBUG "metric": { 13:11:16 DEBUG "instance": "10.142.0.11:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG [...] 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG "instance": "10.142.0.51:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 1735909876, 13:11:16 DEBUG "0.0005666666666684251" 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909876" 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:11:16 DEBUG ----- output ----- 13:11:16 DEBUG { 13:11:16 DEBUG "status": "success", 13:11:16 DEBUG "data": { 13:11:16 DEBUG "resultType": "vector", 13:11:16 DEBUG "result": [ 13:11:16 DEBUG { 13:11:16 DEBUG "metric": { 13:11:16 DEBUG "instance": "10.142.0.11:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG [...] 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG "instance": "10.142.0.51:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 1735909876, 13:11:16 DEBUG "1590737.033333333" 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909876" 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:11:16 DEBUG ----- output ----- 13:11:16 DEBUG { 13:11:16 DEBUG "status": "success", 13:11:16 DEBUG "data": { 13:11:16 DEBUG "resultType": "vector", 13:11:16 DEBUG "result": [ 13:11:16 DEBUG { 13:11:16 DEBUG "metric": { 13:11:16 DEBUG "instance": "10.142.0.11:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG [...] 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG "instance": "10.142.0.51:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 1735909876, 13:11:16 DEBUG "1608315.1000000003" 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909876" 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:11:16 DEBUG ----- output ----- 13:11:16 DEBUG { 13:11:16 DEBUG "status": "success", 13:11:16 DEBUG "data": { 13:11:16 DEBUG "resultType": "vector", 13:11:16 DEBUG "result": [ 13:11:16 DEBUG { 13:11:16 DEBUG "metric": { 13:11:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG [...] 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 1735909876, 13:11:16 DEBUG "0.20413086382235146" 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:11:16 DEBUG ----- output ----- 13:11:16 DEBUG { 13:11:16 DEBUG "status": "success", 13:11:16 DEBUG "data": { 13:11:16 DEBUG "resultType": "vector", 13:11:16 DEBUG "result": [ 13:11:16 DEBUG { 13:11:16 DEBUG "metric": { 13:11:16 DEBUG "instance": "10.142.0.11:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG [...] 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG "instance": "10.142.0.51:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 1735909876, 13:11:16 DEBUG "29900.799999999996" 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909876" 13:11:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909876" 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:11:16 DEBUG ----- output ----- 13:11:16 DEBUG { 13:11:16 DEBUG "status": "success", 13:11:16 DEBUG "data": { 13:11:16 DEBUG "resultType": "vector", 13:11:16 DEBUG "result": [ 13:11:16 DEBUG { 13:11:16 DEBUG "metric": { 13:11:16 DEBUG "pod": "ds-idrepo-0" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG [...] 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 1735909876, 13:11:16 DEBUG "758779904" 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 INFO [http_cmd]: http status code OK 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG --- status code --- 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG --- http response --- 13:11:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:11:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:11:16 DEBUG ----- output ----- 13:11:16 DEBUG ----- output ----- 13:11:16 DEBUG { 13:11:16 DEBUG { 13:11:16 DEBUG "status": "success", 13:11:16 DEBUG "status": "success", 13:11:16 DEBUG "data": { 13:11:16 DEBUG "data": { 13:11:16 DEBUG "resultType": "vector", 13:11:16 DEBUG "resultType": "vector", 13:11:16 DEBUG "result": [ 13:11:16 DEBUG "result": [ 13:11:16 DEBUG { 13:11:16 DEBUG { 13:11:16 DEBUG "metric": { 13:11:16 DEBUG "metric": { 13:11:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:11:16 DEBUG "instance": "10.142.0.11:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG [...] 13:11:16 DEBUG [...] 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG 13:11:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:11:16 DEBUG "instance": "10.142.0.51:9100" 13:11:16 DEBUG }, 13:11:16 DEBUG }, 13:11:16 DEBUG "value": [ 13:11:16 DEBUG "value": [ 13:11:16 DEBUG 1735909876, 13:11:16 DEBUG 1735909876, 13:11:16 DEBUG "14985875456" 13:11:16 DEBUG "0" 13:11:16 DEBUG ] 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:16 DEBUG ] 13:11:16 DEBUG ] 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:16 DEBUG } 13:11:24 INFO 13:11:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:11:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:11:24 INFO [loop_until]: OK (rc = 0) 13:11:24 DEBUG --- stdout --- 13:11:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 145m 0% 3843Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 124m 0% 19052Mi 66% 13:11:24 DEBUG --- stderr --- 13:11:24 DEBUG 13:11:27 INFO 13:11:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:11:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:11:28 INFO [loop_until]: OK (rc = 0) 13:11:28 DEBUG --- stdout --- 13:11:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 124m 13473Mi rcs-645c86fdd6-j4rsk 88m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:11:28 DEBUG --- stderr --- 13:11:28 DEBUG 13:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909934" 13:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909934" 13:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909934" 13:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909934" 13:12:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909934" 13:12:14 INFO [http_cmd]: http status code OK 13:12:14 DEBUG --- status code --- 13:12:14 INFO [http_cmd]: http status code OK 13:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:14 DEBUG --- status code --- 13:12:14 DEBUG --- http response --- 13:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:14 DEBUG --- http response --- 13:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:14 INFO [http_cmd]: http status code OK 13:12:14 DEBUG --- status code --- 13:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:14 DEBUG --- http response --- 13:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:14 INFO [http_cmd]: http status code OK 13:12:14 DEBUG --- status code --- 13:12:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:14 DEBUG --- http response --- 13:12:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:14 ERROR [http_cmd]: ERROR 13:12:14 ERROR --- status code --- 13:12:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:12:14 ERROR --- http response --- 13:12:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:12:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909935" 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 DEBUG --- status code --- 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 INFO [http_cmd]: http status code OK 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG --- status code --- 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG --- http response --- 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "pod": "ds-cts-0" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "0" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "pod": "ds-cts-0" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "0.00023290496535816684" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909936" 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "pod": "ds-cts-0" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "511.44819091608935" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "instance": "10.142.0.11:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "instance": "10.142.0.51:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "0.0011666666666656055" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909936" 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "instance": "10.142.0.11:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "instance": "10.142.0.51:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "76671.40000000001" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909936" 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "instance": "10.142.0.11:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "instance": "10.142.0.51:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "93462.13333333335" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909936" 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "0.03965216699648848" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "instance": "10.142.0.11:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "instance": "10.142.0.51:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "24712.533333333333" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909936" 13:12:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909936" 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "pod": "ds-idrepo-0" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "758779904" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "instance": "10.142.0.11:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "instance": "10.142.0.51:9100" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "0.0006666666666660604" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:16 INFO [http_cmd]: http status code OK 13:12:16 DEBUG --- status code --- 13:12:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:16 DEBUG --- http response --- 13:12:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:12:16 DEBUG ----- output ----- 13:12:16 DEBUG { 13:12:16 DEBUG "status": "success", 13:12:16 DEBUG "data": { 13:12:16 DEBUG "resultType": "vector", 13:12:16 DEBUG "result": [ 13:12:16 DEBUG { 13:12:16 DEBUG "metric": { 13:12:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG [...] 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG 13:12:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:12:16 DEBUG }, 13:12:16 DEBUG "value": [ 13:12:16 DEBUG 1735909936, 13:12:16 DEBUG "14985138176" 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG ] 13:12:16 DEBUG } 13:12:16 DEBUG } 13:12:24 INFO 13:12:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:12:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:12:24 INFO [loop_until]: OK (rc = 0) 13:12:24 DEBUG --- stdout --- 13:12:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 158m 0% 3855Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 125m 0% 19056Mi 67% 13:12:24 DEBUG --- stderr --- 13:12:24 DEBUG 13:12:28 INFO 13:12:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:12:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:12:28 INFO [loop_until]: OK (rc = 0) 13:12:28 DEBUG --- stdout --- 13:12:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 56m 13474Mi rcs-645c86fdd6-j4rsk 18m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:12:28 DEBUG --- stderr --- 13:12:28 DEBUG 13:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735909994" 13:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909994" 13:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909994" 13:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909994" 13:13:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909994" 13:13:14 ERROR [http_cmd]: ERROR 13:13:14 INFO [http_cmd]: http status code OK 13:13:14 ERROR --- status code --- 13:13:14 DEBUG --- status code --- 13:13:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:14 ERROR --- http response --- 13:13:14 DEBUG --- http response --- 13:13:14 INFO [http_cmd]: http status code OK 13:13:14 INFO [http_cmd]: http status code OK 13:13:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:13:14 INFO [http_cmd]: http status code OK 13:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:14 DEBUG --- status code --- 13:13:14 DEBUG --- status code --- 13:13:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:13:14 DEBUG --- status code --- 13:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:14 DEBUG --- http response --- 13:13:14 DEBUG --- http response --- 13:13:14 DEBUG --- http response --- 13:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909995" 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG --- http response --- 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG --- status code --- 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG --- http response --- 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG --- http response --- 13:13:15 INFO [http_cmd]: http status code OK 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 DEBUG --- status code --- 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:15 DEBUG --- http response --- 13:13:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG --- status code --- 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "pod": "ds-cts-0" 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG }, 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG "value": [ 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG 13:13:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "0" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "pod": "ds-cts-0" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "0.0004309236407175939" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909996" 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "pod": "ds-cts-0" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "431.4503134617577" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "instance": "10.142.0.11:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "instance": "10.142.0.51:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "0.000333333333325451" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909996" 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "instance": "10.142.0.11:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "instance": "10.142.0.51:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "80868.63333333335" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909996" 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "instance": "10.142.0.11:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "instance": "10.142.0.51:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "95505.43333333333" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909996" 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "0.09017931862604496" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "instance": "10.142.0.11:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "instance": "10.142.0.51:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "18158.93333333333" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735909996" 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735909996" 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "pod": "ds-idrepo-0" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735909996" 13:13:16 DEBUG 13:13:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "758779904" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "instance": "10.142.0.11:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "instance": "10.142.0.51:9100" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "0.0003333333333330302" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:16 INFO [http_cmd]: http status code OK 13:13:16 DEBUG --- status code --- 13:13:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:16 DEBUG --- http response --- 13:13:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:13:16 DEBUG ----- output ----- 13:13:16 DEBUG { 13:13:16 DEBUG "status": "success", 13:13:16 DEBUG "data": { 13:13:16 DEBUG "resultType": "vector", 13:13:16 DEBUG "result": [ 13:13:16 DEBUG { 13:13:16 DEBUG "metric": { 13:13:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG [...] 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG 13:13:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:13:16 DEBUG }, 13:13:16 DEBUG "value": [ 13:13:16 DEBUG 1735909996, 13:13:16 DEBUG "14986342400" 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG ] 13:13:16 DEBUG } 13:13:16 DEBUG } 13:13:24 INFO 13:13:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:13:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:13:24 INFO [loop_until]: OK (rc = 0) 13:13:24 DEBUG --- stdout --- 13:13:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 115m 0% 3845Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 83m 0% 19053Mi 66% 13:13:24 DEBUG --- stderr --- 13:13:24 DEBUG 13:13:28 INFO 13:13:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:13:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:13:28 INFO [loop_until]: OK (rc = 0) 13:13:28 DEBUG --- stdout --- 13:13:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13472Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:13:28 DEBUG --- stderr --- 13:13:28 DEBUG 13:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910054" 13:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910054" 13:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910054" 13:14:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910054" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910055" 13:14:15 ERROR [http_cmd]: ERROR 13:14:15 ERROR --- status code --- 13:14:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:14:15 ERROR --- http response --- 13:14:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:14:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910055" 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 INFO [http_cmd]: http status code OK 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- status code --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 DEBUG --- http response --- 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG --- http response --- 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735910056, "0" ] } ] } } 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:16 DEBUG ----- output ----- 13:14:16 DEBUG { 13:14:16 DEBUG "status": "success", 13:14:16 DEBUG "data": { 13:14:16 DEBUG "resultType": "vector", 13:14:16 DEBUG "result": [ 13:14:16 DEBUG { 13:14:16 DEBUG "metric": { 13:14:16 DEBUG "pod": "ds-cts-0" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG [...] 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 1735910056, 13:14:16 DEBUG "0" 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG } 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910056" 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:14:16 DEBUG ----- output ----- 13:14:16 DEBUG { 13:14:16 DEBUG "status": "success", 13:14:16 DEBUG "data": { 13:14:16 DEBUG "resultType": "vector", 13:14:16 DEBUG "result": [ 13:14:16 DEBUG { 13:14:16 DEBUG "metric": { 13:14:16 DEBUG "instance": "10.142.0.11:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG [...] 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG "instance": "10.142.0.51:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 1735910056, 13:14:16 DEBUG "0.0002666666666717295" 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG ] 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG } 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG ] 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG } 13:14:16 DEBUG } 13:14:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735910056, "811.7623396249027" ] } ] } } 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910056" 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:14:16 DEBUG ----- output ----- 13:14:16 DEBUG { 13:14:16 DEBUG "status": "success", 13:14:16 DEBUG "data": { 13:14:16 DEBUG "resultType": "vector", 13:14:16 DEBUG "result": [ 13:14:16 DEBUG { 13:14:16 DEBUG "metric": { 13:14:16 DEBUG "instance": "10.142.0.11:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG [...] 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG "instance": "10.142.0.51:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 1735910056, 13:14:16 DEBUG "77748.36666666667" 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG } 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910056" 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:14:16 DEBUG ----- output ----- 13:14:16 DEBUG { 13:14:16 DEBUG "status": "success", 13:14:16 DEBUG "data": { 13:14:16 DEBUG "resultType": "vector", 13:14:16 DEBUG "result": [ 13:14:16 DEBUG { 13:14:16 DEBUG "metric": { 13:14:16 DEBUG "instance": "10.142.0.11:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG [...] 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG "instance": "10.142.0.51:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 1735910056, 13:14:16 DEBUG "95132.43333333332" 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG } 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910056" 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:14:16 DEBUG ----- output ----- 13:14:16 DEBUG { 13:14:16 DEBUG "status": "success", 13:14:16 DEBUG "data": { 13:14:16 DEBUG "resultType": "vector", 13:14:16 DEBUG "result": [ 13:14:16 DEBUG { 13:14:16 DEBUG "metric": { 13:14:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG [...] 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 1735910056, 13:14:16 DEBUG "0.7494483728815371" 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG } 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:14:16 DEBUG ----- output ----- 13:14:16 DEBUG { 13:14:16 DEBUG "status": "success", 13:14:16 DEBUG "data": { 13:14:16 DEBUG "resultType": "vector", 13:14:16 DEBUG "result": [ 13:14:16 DEBUG { 13:14:16 DEBUG "metric": { 13:14:16 DEBUG "instance": "10.142.0.11:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG [...] 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG "instance": "10.142.0.51:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 1735910056, 13:14:16 DEBUG "9147.733333333334" 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG } 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910056" 13:14:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910056" 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:14:16 DEBUG ----- output ----- 13:14:16 DEBUG { 13:14:16 DEBUG "status": "success", 13:14:16 DEBUG "data": { 13:14:16 DEBUG "resultType": "vector", 13:14:16 DEBUG "result": [ 13:14:16 DEBUG { 13:14:16 DEBUG "metric": { 13:14:16 DEBUG "pod": "ds-idrepo-0" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG [...] 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 1735910056, 13:14:16 DEBUG "758779904" 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG } 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:14:16 DEBUG ----- output ----- 13:14:16 DEBUG { 13:14:16 DEBUG "status": "success", 13:14:16 DEBUG "data": { 13:14:16 DEBUG "resultType": "vector", 13:14:16 DEBUG "result": [ 13:14:16 DEBUG { 13:14:16 DEBUG "metric": { 13:14:16 DEBUG "instance": "10.142.0.11:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG [...] 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG "instance": "10.142.0.51:9100" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 1735910056, 13:14:16 DEBUG "0" 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG } 13:14:16 INFO [http_cmd]: http status code OK 13:14:16 DEBUG --- status code --- 13:14:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:16 DEBUG --- http response --- 13:14:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:14:16 DEBUG ----- output ----- 13:14:16 DEBUG { 13:14:16 DEBUG "status": "success", 13:14:16 DEBUG "data": { 13:14:16 DEBUG "resultType": "vector", 13:14:16 DEBUG "result": [ 13:14:16 DEBUG { 13:14:16 DEBUG "metric": { 13:14:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG [...] 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG 13:14:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:14:16 DEBUG }, 13:14:16 DEBUG "value": [ 13:14:16 DEBUG 1735910056, 13:14:16 DEBUG "14987415552" 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG ] 13:14:16 DEBUG } 13:14:16 DEBUG } 13:14:24 INFO 13:14:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:14:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:14:24 INFO [loop_until]: OK (rc = 0) 13:14:24 DEBUG --- stdout --- 13:14:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 104m 0% 3845Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 78m 0% 19055Mi 67% 13:14:24 DEBUG --- stderr --- 13:14:24 DEBUG 13:14:28 INFO 13:14:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:14:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:14:28 INFO [loop_until]: OK (rc = 0) 13:14:28 DEBUG --- stdout --- 13:14:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13476Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:14:28 DEBUG --- stderr --- 13:14:28 DEBUG 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910115" 13:15:15 ERROR [http_cmd]: ERROR 13:15:15 ERROR --- status code --- 13:15:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:15:15 ERROR --- http response --- 13:15:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:15:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910115" 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- http response --- 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:15 INFO [http_cmd]: http status code OK 13:15:15 DEBUG --- status code --- 13:15:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:15 DEBUG --- http response --- 13:15:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 DEBUG --- http response --- 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:15:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "pod": "ds-cts-0" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "0" 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "pod": "ds-cts-0" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "0.0027570387492295034" 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:15:16 DEBUG "instance": "10.142.0.11:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "pod": "ds-cts-0" 13:15:16 DEBUG "instance": "10.142.0.51:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "525.987993194003" 13:15:16 DEBUG "0.000366666666676944" 13:15:16 DEBUG ] 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910116" 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "instance": "10.142.0.11:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "instance": "10.142.0.51:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "76349.33333333334" 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910116" 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "instance": "10.142.0.11:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "instance": "10.142.0.51:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "93731.73333333334" 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910116" 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "0.17720755757269738" 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "instance": "10.142.0.11:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "instance": "10.142.0.51:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "19660.8" 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910116" 13:15:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910116" 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "pod": "ds-idrepo-0" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "758779904" 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "instance": "10.142.0.11:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "instance": "10.142.0.51:9100" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "0.00033333333333397756" 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:16 INFO [http_cmd]: http status code OK 13:15:16 DEBUG --- status code --- 13:15:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:16 DEBUG --- http response --- 13:15:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:15:16 DEBUG ----- output ----- 13:15:16 DEBUG { 13:15:16 DEBUG "status": "success", 13:15:16 DEBUG "data": { 13:15:16 DEBUG "resultType": "vector", 13:15:16 DEBUG "result": [ 13:15:16 DEBUG { 13:15:16 DEBUG "metric": { 13:15:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG [...] 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG 13:15:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:15:16 DEBUG }, 13:15:16 DEBUG "value": [ 13:15:16 DEBUG 1735910116, 13:15:16 DEBUG "14986055680" 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG ] 13:15:16 DEBUG } 13:15:16 DEBUG } 13:15:24 INFO 13:15:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:15:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:15:24 INFO [loop_until]: OK (rc = 0) 13:15:24 DEBUG --- stdout --- 13:15:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 100m 0% 3851Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 80m 0% 19053Mi 66% 13:15:24 DEBUG --- stderr --- 13:15:24 DEBUG 13:15:28 INFO 13:15:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:15:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:15:28 INFO [loop_until]: OK (rc = 0) 13:15:28 DEBUG --- stdout --- 13:15:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 15m 13476Mi rcs-645c86fdd6-j4rsk 29m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:15:28 DEBUG --- stderr --- 13:15:28 DEBUG 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 ERROR [http_cmd]: ERROR 13:16:15 DEBUG --- status code --- 13:16:15 ERROR --- status code --- 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:16:15 DEBUG --- http response --- 13:16:15 ERROR --- http response --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:16:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910175" 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- status code --- 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG --- http response --- 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- http response --- 13:16:15 INFO [http_cmd]: http status code OK 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG --- status code --- 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG --- http response --- 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG { 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG "data": { 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "result": [ 13:16:16 DEBUG { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG [...] 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG "pod": "ds-cts-0" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG "0" 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG { 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG "data": { 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "result": [ 13:16:16 DEBUG { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG [...] 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG "pod": "ds-cts-0" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG "0.0027817121432974516" 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910176" 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG { 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:16:16 DEBUG "data": { 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG { 13:16:16 DEBUG "result": [ 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG { 13:16:16 DEBUG "data": { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:16:16 DEBUG "result": [ 13:16:16 DEBUG }, 13:16:16 DEBUG { 13:16:16 DEBUG "value": [ 13:16:16 DEBUG "metric": { 13:16:16 DEBUG 13:16:16 DEBUG "instance": "10.142.0.11:9100" 13:16:16 DEBUG 13:16:16 DEBUG }, 13:16:16 DEBUG 13:16:16 DEBUG "value": [ 13:16:16 DEBUG [...] 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG [...] 13:16:16 DEBUG "pod": "ds-cts-0" 13:16:16 DEBUG 13:16:16 DEBUG }, 13:16:16 DEBUG 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG "instance": "10.142.0.51:9100" 13:16:16 DEBUG "568.7703950565854" 13:16:16 DEBUG }, 13:16:16 DEBUG ] 13:16:16 DEBUG "value": [ 13:16:16 DEBUG } 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG ] 13:16:16 DEBUG "0.00036666666666557526" 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910176" 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG { 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG "data": { 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "result": [ 13:16:16 DEBUG { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "instance": "10.142.0.11:9100" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG [...] 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG "instance": "10.142.0.51:9100" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG "657944.1333333332" 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910176" 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG { 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG "data": { 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "result": [ 13:16:16 DEBUG { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "instance": "10.142.0.11:9100" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG [...] 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG "instance": "10.142.0.51:9100" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG "674789.4666666666" 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910176" 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG { 13:16:16 DEBUG { 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG "data": { 13:16:16 DEBUG "data": { 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "result": [ 13:16:16 DEBUG "result": [ 13:16:16 DEBUG { 13:16:16 DEBUG { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "instance": "10.142.0.11:9100" 13:16:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:16:16 DEBUG }, 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG [...] 13:16:16 DEBUG [...] 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG "instance": "10.142.0.51:9100" 13:16:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:16:16 DEBUG }, 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG "23074.13333333333" 13:16:16 DEBUG "0.21986089253933008" 13:16:16 DEBUG ] 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910176" 13:16:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910176" 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG { 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG "data": { 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "result": [ 13:16:16 DEBUG { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "pod": "ds-idrepo-0" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG [...] 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG "758779904" 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG { 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG "data": { 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "result": [ 13:16:16 DEBUG { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "instance": "10.142.0.11:9100" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG [...] 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG "instance": "10.142.0.51:9100" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG "0.0006666666666660604" 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:16 INFO [http_cmd]: http status code OK 13:16:16 DEBUG --- status code --- 13:16:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:16 DEBUG --- http response --- 13:16:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:16:16 DEBUG ----- output ----- 13:16:16 DEBUG { 13:16:16 DEBUG "status": "success", 13:16:16 DEBUG "data": { 13:16:16 DEBUG "resultType": "vector", 13:16:16 DEBUG "result": [ 13:16:16 DEBUG { 13:16:16 DEBUG "metric": { 13:16:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG [...] 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG 13:16:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:16:16 DEBUG }, 13:16:16 DEBUG "value": [ 13:16:16 DEBUG 1735910176, 13:16:16 DEBUG "14983135232" 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG ] 13:16:16 DEBUG } 13:16:16 DEBUG } 13:16:24 INFO 13:16:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:16:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:16:24 INFO [loop_until]: OK (rc = 0) 13:16:24 DEBUG --- stdout --- 13:16:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 100m 0% 3852Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 76m 0% 19052Mi 66% 13:16:24 DEBUG --- stderr --- 13:16:24 DEBUG 13:16:28 INFO 13:16:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:16:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:16:28 INFO [loop_until]: OK (rc = 0) 13:16:28 DEBUG --- stdout --- 13:16:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13476Mi rcs-645c86fdd6-j4rsk 56m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:16:28 DEBUG --- stderr --- 13:16:28 DEBUG 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 ERROR [http_cmd]: ERROR 13:17:15 ERROR --- status code --- 13:17:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:17:15 ERROR --- http response --- 13:17:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:17:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910235" 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG --- status code --- 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG --- status code --- 13:17:15 INFO [http_cmd]: http status code OK 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- status code --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 DEBUG --- http response --- 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "pod": "ds-cts-0" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "0" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "pod": "ds-cts-0" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "0" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910236" 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "pod": "ds-cts-0" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "426.2319181362944" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "instance": "10.142.0.11:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "instance": "10.142.0.51:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "0.001266666666666083" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910236" 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "instance": "10.142.0.11:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "instance": "10.142.0.51:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "777307.9" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910236" 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "instance": "10.142.0.11:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "instance": "10.142.0.51:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "794311.7666666667" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910236" 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "instance": "10.142.0.11:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "instance": "10.142.0.51:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "25531.733333333334" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "0.19522667068636676" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910236" 13:17:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910236" 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "pod": "ds-idrepo-0" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "758788096" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "instance": "10.142.0.11:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "instance": "10.142.0.51:9100" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "0.001000000000000038" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:16 INFO [http_cmd]: http status code OK 13:17:16 DEBUG --- status code --- 13:17:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:16 DEBUG --- http response --- 13:17:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:17:16 DEBUG ----- output ----- 13:17:16 DEBUG { 13:17:16 DEBUG "status": "success", 13:17:16 DEBUG "data": { 13:17:16 DEBUG "resultType": "vector", 13:17:16 DEBUG "result": [ 13:17:16 DEBUG { 13:17:16 DEBUG "metric": { 13:17:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG [...] 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG 13:17:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:17:16 DEBUG }, 13:17:16 DEBUG "value": [ 13:17:16 DEBUG 1735910236, 13:17:16 DEBUG "14982381568" 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG ] 13:17:16 DEBUG } 13:17:16 DEBUG } 13:17:24 INFO 13:17:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:17:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:17:24 INFO [loop_until]: OK (rc = 0) 13:17:24 DEBUG --- stdout --- 13:17:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3843Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 78m 0% 19058Mi 67% 13:17:24 DEBUG --- stderr --- 13:17:24 DEBUG 13:17:28 INFO 13:17:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:17:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:17:28 INFO [loop_until]: OK (rc = 0) 13:17:28 DEBUG --- stdout --- 13:17:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 68m 13477Mi rcs-645c86fdd6-j4rsk 7m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:17:28 DEBUG --- stderr --- 13:17:28 DEBUG 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 ERROR [http_cmd]: ERROR 13:18:15 ERROR --- status code --- 13:18:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:18:15 ERROR --- http response --- 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:18:15 DEBUG --- status code --- 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910295" 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 INFO [http_cmd]: http status code OK 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG --- status code --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:15 DEBUG --- http response --- 13:18:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "result": [ 13:18:16 DEBUG { 13:18:16 DEBUG "metric": { 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG 13:18:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 DEBUG 13:18:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG "pod": "ds-cts-0" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "0" 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "result": [ 13:18:16 DEBUG { 13:18:16 DEBUG "metric": { 13:18:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG "pod": "ds-cts-0" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "0.001568906900217278" 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910296" 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG { 13:18:16 DEBUG { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "data": { 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "result": [ 13:18:16 DEBUG "result": [ 13:18:16 DEBUG { 13:18:16 DEBUG { 13:18:16 DEBUG "metric": { 13:18:16 DEBUG "metric": { 13:18:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:18:16 DEBUG "instance": "10.142.0.11:9100" 13:18:16 DEBUG }, 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG [...] 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG "pod": "ds-cts-0" 13:18:16 DEBUG "instance": "10.142.0.51:9100" 13:18:16 DEBUG }, 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "368.9143861190424" 13:18:16 DEBUG "0.00043333333333066547" 13:18:16 DEBUG ] 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910296" 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "result": [ 13:18:16 DEBUG { 13:18:16 DEBUG "metric": { 13:18:16 DEBUG "instance": "10.142.0.11:9100" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG "instance": "10.142.0.51:9100" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "1428805.3999999997" 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910296" 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "result": [ 13:18:16 DEBUG { 13:18:16 DEBUG "metric": { 13:18:16 DEBUG "instance": "10.142.0.11:9100" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG "instance": "10.142.0.51:9100" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "1444033.3666666667" 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910296" 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "result": [ 13:18:16 DEBUG { 13:18:16 DEBUG "metric": { 13:18:16 DEBUG "instance": "10.142.0.11:9100" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG "instance": "10.142.0.51:9100" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "25258.666666666664" 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "result": [ 13:18:16 DEBUG { 13:18:16 DEBUG "metric": { 13:18:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "0.14435098388682058" 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910296" 13:18:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910296" 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 INFO [http_cmd]: http status code OK 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- status code --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:18:16 DEBUG --- http response --- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG { 13:18:16 DEBUG ----- output ----- 13:18:16 DEBUG { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "status": "success", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "data": { 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "result": [ 13:18:16 DEBUG "resultType": "vector", 13:18:16 DEBUG "result": [ 13:18:16 DEBUG { 13:18:16 DEBUG "result": [ 13:18:16 DEBUG { 13:18:16 DEBUG "metric": { 13:18:16 DEBUG { 13:18:16 DEBUG "metric": { 13:18:16 DEBUG "instance": "10.142.0.11:9100" 13:18:16 DEBUG "metric": { 13:18:16 DEBUG "pod": "ds-idrepo-0" 13:18:16 DEBUG }, 13:18:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG [...] 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG 13:18:16 DEBUG "instance": "10.142.0.51:9100" 13:18:16 DEBUG 13:18:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:18:16 DEBUG }, 13:18:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG }, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "value": [ 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "0" 13:18:16 DEBUG 1735910296, 13:18:16 DEBUG "14986067968" 13:18:16 DEBUG ] 13:18:16 DEBUG "758779904" 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG ] 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:16 DEBUG } 13:18:24 INFO 13:18:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:18:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:18:24 INFO [loop_until]: OK (rc = 0) 13:18:24 DEBUG --- stdout --- 13:18:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 112m 0% 3852Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 89m 0% 19057Mi 67% 13:18:24 DEBUG --- stderr --- 13:18:24 DEBUG 13:18:28 INFO 13:18:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:18:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:18:28 INFO [loop_until]: OK (rc = 0) 13:18:28 DEBUG --- stdout --- 13:18:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 55m 13477Mi rcs-645c86fdd6-j4rsk 93m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:18:28 DEBUG --- stderr --- 13:18:28 DEBUG 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 ERROR [http_cmd]: ERROR 13:19:15 DEBUG --- status code --- 13:19:15 ERROR --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 ERROR --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:19:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910355" 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG --- status code --- 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 INFO [http_cmd]: http status code OK 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG --- status code --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG --- http response --- 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- http response --- 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:19:16 DEBUG ----- output ----- 13:19:16 DEBUG { 13:19:16 DEBUG "status": "success", 13:19:16 DEBUG "data": { 13:19:16 DEBUG "resultType": "vector", 13:19:16 DEBUG "result": [ 13:19:16 DEBUG { 13:19:16 DEBUG "metric": { 13:19:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG [...] 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG "pod": "ds-cts-0" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 1735910356, 13:19:16 DEBUG "0" 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:19:16 DEBUG ----- output ----- 13:19:16 DEBUG { 13:19:16 DEBUG "status": "success", 13:19:16 DEBUG "data": { 13:19:16 DEBUG "resultType": "vector", 13:19:16 DEBUG "result": [ 13:19:16 DEBUG { 13:19:16 DEBUG "metric": { 13:19:16 DEBUG "instance": "10.142.0.11:9100" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG [...] 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG "instance": "10.142.0.51:9100" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 1735910356, 13:19:16 DEBUG "0.0003666666666693648" 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910356" 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:19:16 DEBUG ----- output ----- 13:19:16 DEBUG { 13:19:16 DEBUG "status": "success", 13:19:16 DEBUG "data": { 13:19:16 DEBUG "resultType": "vector", 13:19:16 DEBUG "result": [ 13:19:16 DEBUG { 13:19:16 DEBUG "metric": { 13:19:16 DEBUG "instance": "10.142.0.11:9100" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG [...] 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG "instance": "10.142.0.51:9100" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 1735910356, 13:19:16 DEBUG "2356440.0333333337" 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910356" 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:19:16 DEBUG ----- output ----- 13:19:16 DEBUG { 13:19:16 DEBUG "status": "success", 13:19:16 DEBUG "data": { 13:19:16 DEBUG "resultType": "vector", 13:19:16 DEBUG "result": [ 13:19:16 DEBUG { 13:19:16 DEBUG "metric": { 13:19:16 DEBUG "instance": "10.142.0.11:9100" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG [...] 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG "instance": "10.142.0.51:9100" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 1735910356, 13:19:16 DEBUG "2373384.2333333334" 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910356" 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:19:16 DEBUG ----- output ----- 13:19:16 DEBUG { 13:19:16 DEBUG "status": "success", 13:19:16 DEBUG "data": { 13:19:16 DEBUG "resultType": "vector", 13:19:16 DEBUG "result": [ 13:19:16 DEBUG { 13:19:16 DEBUG "metric": { 13:19:16 DEBUG "instance": "10.142.0.11:9100" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG [...] 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG "instance": "10.142.0.51:9100" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 1735910356, 13:19:16 DEBUG "21026.13333333333" 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:19:16 DEBUG ----- output ----- 13:19:16 DEBUG { 13:19:16 DEBUG "status": "success", 13:19:16 DEBUG "data": { 13:19:16 DEBUG "resultType": "vector", 13:19:16 DEBUG "result": [ 13:19:16 DEBUG { 13:19:16 DEBUG "metric": { 13:19:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG [...] 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 1735910356, 13:19:16 DEBUG "0.42892726963849825" 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910356" 13:19:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910356" 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:19:16 INFO [http_cmd]: http status code OK 13:19:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:19:16 DEBUG ----- output ----- 13:19:16 DEBUG --- status code --- 13:19:16 DEBUG ----- output ----- 13:19:16 DEBUG { 13:19:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:16 DEBUG { 13:19:16 DEBUG "status": "success", 13:19:16 DEBUG --- http response --- 13:19:16 DEBUG "status": "success", 13:19:16 DEBUG "data": { 13:19:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:19:16 DEBUG "data": { 13:19:16 DEBUG "resultType": "vector", 13:19:16 DEBUG ----- output ----- 13:19:16 DEBUG "resultType": "vector", 13:19:16 DEBUG "result": [ 13:19:16 DEBUG { 13:19:16 DEBUG "result": [ 13:19:16 DEBUG { 13:19:16 DEBUG "status": "success", 13:19:16 DEBUG { 13:19:16 DEBUG "metric": { 13:19:16 DEBUG "data": { 13:19:16 DEBUG "metric": { 13:19:16 DEBUG "instance": "10.142.0.11:9100" 13:19:16 DEBUG "resultType": "vector", 13:19:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:19:16 DEBUG }, 13:19:16 DEBUG "result": [ 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG { 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 13:19:16 DEBUG "metric": { 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG "pod": "ds-idrepo-0" 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG [...] 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG [...] 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG [...] 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG 13:19:16 DEBUG "instance": "10.142.0.51:9100" 13:19:16 DEBUG 13:19:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:19:16 DEBUG }, 13:19:16 DEBUG 13:19:16 DEBUG }, 13:19:16 DEBUG "value": [ 13:19:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:19:16 DEBUG "value": [ 13:19:16 DEBUG 1735910356, 13:19:16 DEBUG }, 13:19:16 DEBUG 1735910356, 13:19:16 DEBUG "0" 13:19:16 DEBUG "value": [ 13:19:16 DEBUG "14991532032" 13:19:16 DEBUG ] 13:19:16 DEBUG 1735910356, 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG "758779904" 13:19:16 DEBUG } 13:19:16 DEBUG ] 13:19:16 DEBUG ] 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 DEBUG ] 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:16 DEBUG } 13:19:24 INFO 13:19:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:19:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:19:24 INFO [loop_until]: OK (rc = 0) 13:19:24 DEBUG --- stdout --- 13:19:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 158m 0% 3856Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 456m 1% 19058Mi 67% 13:19:24 DEBUG --- stderr --- 13:19:24 DEBUG 13:19:28 INFO 13:19:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:19:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:19:28 INFO [loop_until]: OK (rc = 0) 13:19:28 DEBUG --- stdout --- 13:19:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 297m 13477Mi rcs-645c86fdd6-j4rsk 140m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:19:28 DEBUG --- stderr --- 13:19:28 DEBUG 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910415" 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 ERROR [http_cmd]: ERROR 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 ERROR --- status code --- 13:20:15 DEBUG --- status code --- 13:20:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 ERROR --- http response --- 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910415" 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:15 INFO [http_cmd]: http status code OK 13:20:15 DEBUG --- status code --- 13:20:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:15 DEBUG --- http response --- 13:20:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG { 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "data": { 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "result": [ 13:20:16 DEBUG { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG [...] 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG "pod": "ds-idrepo-0" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG "409455.0493392908" 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG { 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "data": { 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "result": [ 13:20:16 DEBUG { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG [...] 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG "pod": "ds-idrepo-0" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG "0.02013063599654176" 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG { 13:20:16 DEBUG { 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "data": { 13:20:16 DEBUG "data": { 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "result": [ 13:20:16 DEBUG "result": [ 13:20:16 DEBUG { 13:20:16 DEBUG { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:20:16 DEBUG "instance": "10.142.0.11:9100" 13:20:16 DEBUG }, 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG [...] 13:20:16 DEBUG [...] 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG "pod": "ds-idrepo-0" 13:20:16 DEBUG "instance": "10.142.0.51:9100" 13:20:16 DEBUG }, 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG "314.4816047152772" 13:20:16 DEBUG "0.00026666666666793997" 13:20:16 DEBUG ] 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910416" 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG { 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "data": { 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "result": [ 13:20:16 DEBUG { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "instance": "10.142.0.11:9100" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG [...] 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG "instance": "10.142.0.51:9100" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG "2370092.5666666673" 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910416" 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG { 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "data": { 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "result": [ 13:20:16 DEBUG { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "instance": "10.142.0.11:9100" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG [...] 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG "instance": "10.142.0.51:9100" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG "2386842.1666666674" 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910416" 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG { 13:20:16 DEBUG { 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "data": { 13:20:16 DEBUG "data": { 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "result": [ 13:20:16 DEBUG "result": [ 13:20:16 DEBUG { 13:20:16 DEBUG { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:20:16 DEBUG "instance": "10.142.0.11:9100" 13:20:16 DEBUG }, 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG [...] 13:20:16 DEBUG [...] 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:20:16 DEBUG "instance": "10.142.0.51:9100" 13:20:16 DEBUG }, 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG "0.09756502692046672" 13:20:16 DEBUG "7236.266666666666" 13:20:16 DEBUG ] 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910416" 13:20:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910416" 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 INFO [http_cmd]: http status code OK 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- status code --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:20:16 DEBUG --- http response --- 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG { 13:20:16 DEBUG ----- output ----- 13:20:16 DEBUG { 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG { 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "data": { 13:20:16 DEBUG "status": "success", 13:20:16 DEBUG "data": { 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "data": { 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "result": [ 13:20:16 DEBUG "resultType": "vector", 13:20:16 DEBUG "result": [ 13:20:16 DEBUG { 13:20:16 DEBUG "result": [ 13:20:16 DEBUG { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG { 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "instance": "10.142.0.11:9100" 13:20:16 DEBUG "metric": { 13:20:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:20:16 DEBUG }, 13:20:16 DEBUG "pod": "ds-idrepo-0" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG [...] 13:20:16 DEBUG 13:20:16 DEBUG [...] 13:20:16 DEBUG 13:20:16 DEBUG [...] 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG 13:20:16 DEBUG "instance": "10.142.0.51:9100" 13:20:16 DEBUG 13:20:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:20:16 DEBUG }, 13:20:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG }, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG "value": [ 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG "0" 13:20:16 DEBUG 1735910416, 13:20:16 DEBUG "14989553664" 13:20:16 DEBUG ] 13:20:16 DEBUG "758779904" 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG ] 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:16 DEBUG } 13:20:24 INFO 13:20:24 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:20:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:20:25 INFO [loop_until]: OK (rc = 0) 13:20:25 DEBUG --- stdout --- 13:20:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 191m 1% 3859Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 177m 0% 19064Mi 67% 13:20:25 DEBUG --- stderr --- 13:20:25 DEBUG 13:20:28 INFO 13:20:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:20:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:20:28 INFO [loop_until]: OK (rc = 0) 13:20:28 DEBUG --- stdout --- 13:20:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 128m 13476Mi rcs-645c86fdd6-j4rsk 120m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:20:28 DEBUG --- stderr --- 13:20:28 DEBUG 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 ERROR [http_cmd]: ERROR 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 DEBUG --- status code --- 13:21:15 DEBUG --- status code --- 13:21:15 DEBUG --- status code --- 13:21:15 ERROR --- status code --- 13:21:15 DEBUG --- status code --- 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG --- http response --- 13:21:15 DEBUG --- http response --- 13:21:15 DEBUG --- http response --- 13:21:15 ERROR --- http response --- 13:21:15 DEBUG --- http response --- 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 DEBUG --- status code --- 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG --- http response --- 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 DEBUG --- status code --- 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG --- http response --- 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 DEBUG --- status code --- 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG --- http response --- 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910475" 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 DEBUG --- status code --- 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG --- http response --- 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 DEBUG --- status code --- 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG --- http response --- 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 DEBUG --- status code --- 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG --- http response --- 13:21:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 DEBUG --- status code --- 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 INFO [http_cmd]: http status code OK 13:21:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:15 DEBUG --- status code --- 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG { 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG "data": { 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "result": [ 13:21:16 DEBUG { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "pod": "ds-cts-0" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG [...] 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG "0" 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG { 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG "data": { 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "result": [ 13:21:16 DEBUG { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "pod": "ds-cts-0" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG [...] 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG "0" 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910476" 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG { 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:21:16 DEBUG "data": { 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG { 13:21:16 DEBUG "result": [ 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG { 13:21:16 DEBUG "data": { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "pod": "ds-cts-0" 13:21:16 DEBUG "result": [ 13:21:16 DEBUG }, 13:21:16 DEBUG { 13:21:16 DEBUG "value": [ 13:21:16 DEBUG "metric": { 13:21:16 DEBUG 13:21:16 DEBUG "instance": "10.142.0.11:9100" 13:21:16 DEBUG 13:21:16 DEBUG }, 13:21:16 DEBUG 13:21:16 DEBUG "value": [ 13:21:16 DEBUG [...] 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG [...] 13:21:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:21:16 DEBUG 13:21:16 DEBUG }, 13:21:16 DEBUG 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG "instance": "10.142.0.51:9100" 13:21:16 DEBUG "278.90507966771077" 13:21:16 DEBUG }, 13:21:16 DEBUG ] 13:21:16 DEBUG "value": [ 13:21:16 DEBUG } 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG ] 13:21:16 DEBUG "0.00043333333333066547" 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910476" 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG { 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG "data": { 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "result": [ 13:21:16 DEBUG { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "instance": "10.142.0.11:9100" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG [...] 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG "instance": "10.142.0.51:9100" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG "1880465.2999999998" 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910476" 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG { 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG "data": { 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "result": [ 13:21:16 DEBUG { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "instance": "10.142.0.11:9100" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG [...] 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG "instance": "10.142.0.51:9100" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG "1897394.1666666667" 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910476" 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG { 13:21:16 DEBUG { 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG "data": { 13:21:16 DEBUG "data": { 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "result": [ 13:21:16 DEBUG "result": [ 13:21:16 DEBUG { 13:21:16 DEBUG { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:21:16 DEBUG "instance": "10.142.0.11:9100" 13:21:16 DEBUG }, 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG [...] 13:21:16 DEBUG [...] 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:21:16 DEBUG "instance": "10.142.0.51:9100" 13:21:16 DEBUG }, 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG "0.09564262131050419" 13:21:16 DEBUG "26350.933333333334" 13:21:16 DEBUG ] 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910476" 13:21:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910476" 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG { 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG "data": { 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "result": [ 13:21:16 DEBUG { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "instance": "10.142.0.11:9100" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG [...] 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG "instance": "10.142.0.51:9100" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG "0.0003333333333330302" 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG { 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG "data": { 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "result": [ 13:21:16 DEBUG { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "pod": "ds-idrepo-0" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG [...] 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:21:16 INFO [http_cmd]: http status code OK 13:21:16 DEBUG }, 13:21:16 DEBUG --- status code --- 13:21:16 DEBUG "value": [ 13:21:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG --- http response --- 13:21:16 DEBUG "758779904" 13:21:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:21:16 DEBUG ] 13:21:16 DEBUG ----- output ----- 13:21:16 DEBUG } 13:21:16 DEBUG { 13:21:16 DEBUG ] 13:21:16 DEBUG "status": "success", 13:21:16 DEBUG } 13:21:16 DEBUG "data": { 13:21:16 DEBUG } 13:21:16 DEBUG "resultType": "vector", 13:21:16 DEBUG "result": [ 13:21:16 DEBUG { 13:21:16 DEBUG "metric": { 13:21:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG [...] 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG 13:21:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:21:16 DEBUG }, 13:21:16 DEBUG "value": [ 13:21:16 DEBUG 1735910476, 13:21:16 DEBUG "14991642624" 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG ] 13:21:16 DEBUG } 13:21:16 DEBUG } 13:21:25 INFO 13:21:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:21:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:21:25 INFO [loop_until]: OK (rc = 0) 13:21:25 DEBUG --- stdout --- 13:21:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 220m 1% 3842Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 189m 0% 19057Mi 67% 13:21:25 DEBUG --- stderr --- 13:21:25 DEBUG 13:21:28 INFO 13:21:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:21:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:21:28 INFO [loop_until]: OK (rc = 0) 13:21:28 DEBUG --- stdout --- 13:21:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 129m 13478Mi rcs-645c86fdd6-j4rsk 151m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:21:28 DEBUG --- stderr --- 13:21:28 DEBUG 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910535" 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910535" 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910535" 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910535" 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910535" 13:22:15 INFO [http_cmd]: http status code OK 13:22:15 ERROR [http_cmd]: ERROR 13:22:15 DEBUG --- status code --- 13:22:15 ERROR --- status code --- 13:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:22:15 DEBUG --- http response --- 13:22:15 ERROR --- http response --- 13:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:22:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:22:15 INFO [http_cmd]: http status code OK 13:22:15 DEBUG --- status code --- 13:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:15 INFO [http_cmd]: http status code OK 13:22:15 DEBUG --- http response --- 13:22:15 INFO [http_cmd]: http status code OK 13:22:15 DEBUG --- status code --- 13:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:15 DEBUG --- status code --- 13:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:15 DEBUG --- http response --- 13:22:15 DEBUG --- http response --- 13:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910535" 13:22:15 INFO [http_cmd]: http status code OK 13:22:15 DEBUG --- status code --- 13:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:15 DEBUG --- http response --- 13:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910535" 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910535" 13:22:15 INFO [http_cmd]: http status code OK 13:22:15 DEBUG --- status code --- 13:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:15 DEBUG --- http response --- 13:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:15 INFO [http_cmd]: http status code OK 13:22:15 DEBUG --- status code --- 13:22:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:15 DEBUG --- http response --- 13:22:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910535" 13:22:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910535" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "pod": "ds-idrepo-0" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "0" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "pod": "ds-cts-0" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "0" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910536" 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "pod": "ds-idrepo-0" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "655.9361097770154" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "instance": "10.142.0.11:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "instance": "10.142.0.51:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "0.0011666666666731848" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910536" 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "instance": "10.142.0.11:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "instance": "10.142.0.51:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "1221037.0666666667" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910536" 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "instance": "10.142.0.11:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "instance": "10.142.0.51:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "1238491.5333333334" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910536" 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "instance": "10.142.0.11:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "instance": "10.142.0.51:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "24576" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "0.04062926867779441" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910536" 13:22:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910536" 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "instance": "10.142.0.11:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "instance": "10.142.0.51:9100" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "0.001333333333333068" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG { 13:22:16 DEBUG "status": "success", 13:22:16 INFO [http_cmd]: http status code OK 13:22:16 DEBUG "data": { 13:22:16 DEBUG --- status code --- 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:16 DEBUG "result": [ 13:22:16 DEBUG --- http response --- 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "pod": "ds-idrepo-0" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "758779904" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:22:16 DEBUG ] 13:22:16 DEBUG ----- output ----- 13:22:16 DEBUG } 13:22:16 DEBUG { 13:22:16 DEBUG } 13:22:16 DEBUG "status": "success", 13:22:16 DEBUG "data": { 13:22:16 DEBUG "resultType": "vector", 13:22:16 DEBUG "result": [ 13:22:16 DEBUG { 13:22:16 DEBUG "metric": { 13:22:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG [...] 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG 13:22:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:22:16 DEBUG }, 13:22:16 DEBUG "value": [ 13:22:16 DEBUG 1735910536, 13:22:16 DEBUG "14986559488" 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG ] 13:22:16 DEBUG } 13:22:16 DEBUG } 13:22:25 INFO 13:22:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:22:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:22:25 INFO [loop_until]: OK (rc = 0) 13:22:25 DEBUG --- stdout --- 13:22:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 217m 1% 3842Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 212m 0% 19054Mi 66% 13:22:25 DEBUG --- stderr --- 13:22:25 DEBUG 13:22:28 INFO 13:22:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:22:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:22:28 INFO [loop_until]: OK (rc = 0) 13:22:28 DEBUG --- stdout --- 13:22:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 81m 13477Mi rcs-645c86fdd6-j4rsk 69m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:22:28 DEBUG --- stderr --- 13:22:28 DEBUG 13:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910595" 13:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910595" 13:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910595" 13:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910595" 13:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910595" 13:23:15 INFO [http_cmd]: http status code OK 13:23:15 INFO [http_cmd]: http status code OK 13:23:15 ERROR [http_cmd]: ERROR 13:23:15 INFO [http_cmd]: http status code OK 13:23:15 INFO [http_cmd]: http status code OK 13:23:15 DEBUG --- status code --- 13:23:15 DEBUG --- status code --- 13:23:15 ERROR --- status code --- 13:23:15 DEBUG --- status code --- 13:23:15 DEBUG --- status code --- 13:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:15 DEBUG --- http response --- 13:23:15 DEBUG --- http response --- 13:23:15 ERROR --- http response --- 13:23:15 DEBUG --- http response --- 13:23:15 DEBUG --- http response --- 13:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910595" 13:23:15 INFO [http_cmd]: http status code OK 13:23:15 DEBUG --- status code --- 13:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:15 DEBUG --- http response --- 13:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910595" 13:23:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910595" 13:23:15 INFO [http_cmd]: http status code OK 13:23:15 DEBUG --- status code --- 13:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:15 DEBUG --- http response --- 13:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:15 INFO [http_cmd]: http status code OK 13:23:15 DEBUG --- status code --- 13:23:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:15 DEBUG --- http response --- 13:23:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- status code --- 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "pod": "ds-cts-0" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "0" 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "pod": "ds-cts-0" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "0.0007452370564769417" 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910596" 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "pod": "ds-cts-0" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "595.695171611402" 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "instance": "10.142.0.11:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "instance": "10.142.0.51:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "0.00040000000000190995" 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910596" 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "instance": "10.142.0.11:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "instance": "10.142.0.51:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "1086831.4999999998" 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910596" 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "instance": "10.142.0.11:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "instance": "10.142.0.51:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "1103985.8666666667" 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910596" 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "instance": "10.142.0.11:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "instance": "10.142.0.51:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "20070.399999999998" 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "0.11223755097957872" 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910596" 13:23:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910596" 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "instance": "10.142.0.11:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "instance": "10.142.0.51:9100" 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "0" 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 INFO [http_cmd]: http status code OK 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG --- status code --- 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG --- http response --- 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:23:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG ----- output ----- 13:23:16 DEBUG { 13:23:16 DEBUG { 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "status": "success", 13:23:16 DEBUG "data": { 13:23:16 DEBUG "data": { 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "resultType": "vector", 13:23:16 DEBUG "result": [ 13:23:16 DEBUG "result": [ 13:23:16 DEBUG { 13:23:16 DEBUG { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "metric": { 13:23:16 DEBUG "pod": "ds-idrepo-0" 13:23:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:23:16 DEBUG }, 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG [...] 13:23:16 DEBUG [...] 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG 13:23:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:23:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:23:16 DEBUG }, 13:23:16 DEBUG }, 13:23:16 DEBUG "value": [ 13:23:16 DEBUG "value": [ 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG 1735910596, 13:23:16 DEBUG "758779904" 13:23:16 DEBUG "14984421376" 13:23:16 DEBUG ] 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 DEBUG ] 13:23:16 DEBUG ] 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:16 DEBUG } 13:23:25 INFO 13:23:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:23:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:23:25 INFO [loop_until]: OK (rc = 0) 13:23:25 DEBUG --- stdout --- 13:23:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 158m 0% 3843Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 127m 0% 19054Mi 66% 13:23:25 DEBUG --- stderr --- 13:23:25 DEBUG 13:23:28 INFO 13:23:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:23:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:23:28 INFO [loop_until]: OK (rc = 0) 13:23:28 DEBUG --- stdout --- 13:23:28 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 59m 13478Mi rcs-645c86fdd6-j4rsk 64m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:23:28 DEBUG --- stderr --- 13:23:28 DEBUG 13:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910655" 13:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910655" 13:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910655" 13:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910655" 13:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910655" 13:24:15 INFO [http_cmd]: http status code OK 13:24:15 INFO [http_cmd]: http status code OK 13:24:15 DEBUG --- status code --- 13:24:15 DEBUG --- status code --- 13:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:15 DEBUG --- http response --- 13:24:15 DEBUG --- http response --- 13:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:15 INFO [http_cmd]: http status code OK 13:24:15 DEBUG --- status code --- 13:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:15 INFO [http_cmd]: http status code OK 13:24:15 DEBUG --- http response --- 13:24:15 DEBUG --- status code --- 13:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:15 DEBUG --- http response --- 13:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:15 ERROR [http_cmd]: ERROR 13:24:15 ERROR --- status code --- 13:24:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:24:15 ERROR --- http response --- 13:24:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:24:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910655" 13:24:15 INFO [http_cmd]: http status code OK 13:24:15 DEBUG --- status code --- 13:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:15 DEBUG --- http response --- 13:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910655" 13:24:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910655" 13:24:15 INFO [http_cmd]: http status code OK 13:24:15 DEBUG --- status code --- 13:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:15 DEBUG --- http response --- 13:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:15 INFO [http_cmd]: http status code OK 13:24:15 DEBUG --- status code --- 13:24:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:15 DEBUG --- http response --- 13:24:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG --- status code --- 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG { 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG "data": { 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "pod": "ds-cts-0" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "0" 13:24:16 DEBUG ] 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG ] 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG } 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG } 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "pod": "ds-cts-0" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "0.0026919483374537962" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910656" 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "pod": "ds-cts-0" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "422.0939818631492" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "instance": "10.142.0.11:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "instance": "10.142.0.51:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "0.0005333333333320903" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910656" 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "instance": "10.142.0.11:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "instance": "10.142.0.51:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "81557.59999999999" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910656" 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "instance": "10.142.0.11:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "instance": "10.142.0.51:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "95511.63333333333" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910656" 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "0.08964768628756555" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "instance": "10.142.0.11:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "instance": "10.142.0.51:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "20480" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910656" 13:24:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910656" 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "instance": "10.142.0.11:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "instance": "10.142.0.51:9100" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "0.0003333333333330302" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "14984470528" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:16 INFO [http_cmd]: http status code OK 13:24:16 DEBUG --- status code --- 13:24:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:24:16 DEBUG --- http response --- 13:24:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:24:16 DEBUG ----- output ----- 13:24:16 DEBUG { 13:24:16 DEBUG "status": "success", 13:24:16 DEBUG "data": { 13:24:16 DEBUG "resultType": "vector", 13:24:16 DEBUG "result": [ 13:24:16 DEBUG { 13:24:16 DEBUG "metric": { 13:24:16 DEBUG "pod": "ds-idrepo-0" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG [...] 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG 13:24:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:24:16 DEBUG }, 13:24:16 DEBUG "value": [ 13:24:16 DEBUG 1735910656, 13:24:16 DEBUG "758779904" 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG ] 13:24:16 DEBUG } 13:24:16 DEBUG } 13:24:25 INFO 13:24:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:24:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:24:25 INFO [loop_until]: OK (rc = 0) 13:24:25 DEBUG --- stdout --- 13:24:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 116m 0% 3840Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 75m 0% 19053Mi 66% 13:24:25 DEBUG --- stderr --- 13:24:25 DEBUG 13:24:28 INFO 13:24:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:24:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:24:29 INFO [loop_until]: OK (rc = 0) 13:24:29 DEBUG --- stdout --- 13:24:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13476Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:24:29 DEBUG --- stderr --- 13:24:29 DEBUG 13:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910715" 13:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910715" 13:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910715" 13:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910715" 13:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910715" 13:25:15 INFO [http_cmd]: http status code OK 13:25:15 DEBUG --- status code --- 13:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:15 DEBUG --- http response --- 13:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:15 ERROR [http_cmd]: ERROR 13:25:15 ERROR --- status code --- 13:25:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:25:15 ERROR --- http response --- 13:25:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:25:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:25:15 INFO [http_cmd]: http status code OK 13:25:15 DEBUG --- status code --- 13:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:15 DEBUG --- http response --- 13:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:15 INFO [http_cmd]: http status code OK 13:25:15 DEBUG --- status code --- 13:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:15 DEBUG --- http response --- 13:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:15 INFO [http_cmd]: http status code OK 13:25:15 DEBUG --- status code --- 13:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:15 DEBUG --- http response --- 13:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910715" 13:25:15 INFO [http_cmd]: http status code OK 13:25:15 DEBUG --- status code --- 13:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:15 DEBUG --- http response --- 13:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910715" 13:25:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910715" 13:25:15 INFO [http_cmd]: http status code OK 13:25:15 DEBUG --- status code --- 13:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:15 DEBUG --- http response --- 13:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:15 INFO [http_cmd]: http status code OK 13:25:15 DEBUG --- status code --- 13:25:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:15 DEBUG --- http response --- 13:25:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735910716, "0" ] } ] } } 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:25:16 DEBUG ----- output ----- 13:25:16 DEBUG { 13:25:16 DEBUG "status": "success", 13:25:16 DEBUG "data": { 13:25:16 DEBUG "resultType": "vector", 13:25:16 DEBUG "result": [ 13:25:16 DEBUG { 13:25:16 DEBUG "metric": { 13:25:16 DEBUG "pod": "ds-cts-0" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG [...] 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 1735910716, 13:25:16 DEBUG "0" 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG } 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910716" 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735910716, "443.6581579232409" ] } ] } } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:25:16 DEBUG ----- output ----- 13:25:16 DEBUG { 13:25:16 DEBUG "status": "success", 13:25:16 DEBUG "data": { 13:25:16 DEBUG "resultType": "vector", 13:25:16 DEBUG "result": [ 13:25:16 DEBUG { 13:25:16 DEBUG "metric": { 13:25:16 DEBUG "instance": "10.142.0.11:9100" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG [...] 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG "instance": "10.142.0.51:9100" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 1735910716, 13:25:16 DEBUG "0.0008999999999957707" 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG } 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910716" 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:25:16 DEBUG ----- output ----- 13:25:16 DEBUG { 13:25:16 DEBUG "status": "success", 13:25:16 DEBUG "data": { 13:25:16 DEBUG "resultType": "vector", 13:25:16 DEBUG "result": [ 13:25:16 DEBUG { 13:25:16 DEBUG "metric": { 13:25:16 DEBUG "instance": "10.142.0.11:9100" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG [...] 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG "instance": "10.142.0.51:9100" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 1735910716, 13:25:16 DEBUG "2306930.0000000005" 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG } 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910716" 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:25:16 DEBUG ----- output ----- 13:25:16 DEBUG { 13:25:16 DEBUG "status": "success", 13:25:16 DEBUG "data": { 13:25:16 DEBUG "resultType": "vector", 13:25:16 DEBUG "result": [ 13:25:16 DEBUG { 13:25:16 DEBUG "metric": { 13:25:16 DEBUG "instance": "10.142.0.11:9100" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG [...] 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG "instance": "10.142.0.51:9100" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 1735910716, 13:25:16 DEBUG "2323520.6000000006" 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG } 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910716" 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG ----- output ----- 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG { 13:25:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:25:16 DEBUG "status": "success", 13:25:16 DEBUG ----- output ----- 13:25:16 DEBUG "data": { 13:25:16 DEBUG { 13:25:16 DEBUG "resultType": "vector", 13:25:16 DEBUG "status": "success", 13:25:16 DEBUG "result": [ 13:25:16 DEBUG "data": { 13:25:16 DEBUG { 13:25:16 DEBUG "resultType": "vector", 13:25:16 DEBUG "metric": { 13:25:16 DEBUG "result": [ 13:25:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:25:16 DEBUG { 13:25:16 DEBUG }, 13:25:16 DEBUG "metric": { 13:25:16 DEBUG "value": [ 13:25:16 DEBUG "instance": "10.142.0.11:9100" 13:25:16 DEBUG 13:25:16 DEBUG }, 13:25:16 DEBUG 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG [...] 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG [...] 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:25:16 DEBUG }, 13:25:16 DEBUG 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 13:25:16 DEBUG 1735910716, 13:25:16 DEBUG "instance": "10.142.0.51:9100" 13:25:16 DEBUG "0.5511952499260598" 13:25:16 DEBUG }, 13:25:16 DEBUG ] 13:25:16 DEBUG "value": [ 13:25:16 DEBUG } 13:25:16 DEBUG 1735910716, 13:25:16 DEBUG ] 13:25:16 DEBUG "23483.733333333334" 13:25:16 DEBUG } 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG } 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG } 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910716" 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910716" 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:25:16 DEBUG ----- output ----- 13:25:16 DEBUG { 13:25:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910716" 13:25:16 DEBUG "status": "success", 13:25:16 DEBUG "data": { 13:25:16 DEBUG "resultType": "vector", 13:25:16 DEBUG "result": [ 13:25:16 DEBUG { 13:25:16 DEBUG "metric": { 13:25:16 DEBUG "instance": "10.142.0.11:9100" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG [...] 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG "instance": "10.142.0.51:9100" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 1735910716, 13:25:16 DEBUG "0.00033333333333397756" 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG } 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:25:16 DEBUG ----- output ----- 13:25:16 DEBUG { 13:25:16 DEBUG "status": "success", 13:25:16 DEBUG "data": { 13:25:16 DEBUG "resultType": "vector", 13:25:16 DEBUG "result": [ 13:25:16 DEBUG { 13:25:16 DEBUG "metric": { 13:25:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG [...] 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 INFO [http_cmd]: http status code OK 13:25:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:25:16 DEBUG --- status code --- 13:25:16 DEBUG }, 13:25:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:25:16 DEBUG "value": [ 13:25:16 DEBUG --- http response --- 13:25:16 DEBUG 1735910716, 13:25:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:25:16 DEBUG "14986313728" 13:25:16 DEBUG ----- output ----- 13:25:16 DEBUG ] 13:25:16 DEBUG { 13:25:16 DEBUG } 13:25:16 DEBUG "status": "success", 13:25:16 DEBUG ] 13:25:16 DEBUG "data": { 13:25:16 DEBUG } 13:25:16 DEBUG "resultType": "vector", 13:25:16 DEBUG } 13:25:16 DEBUG "result": [ 13:25:16 DEBUG { 13:25:16 DEBUG "metric": { 13:25:16 DEBUG "pod": "ds-idrepo-0" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG [...] 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG 13:25:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:25:16 DEBUG }, 13:25:16 DEBUG "value": [ 13:25:16 DEBUG 1735910716, 13:25:16 DEBUG "758779904" 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG ] 13:25:16 DEBUG } 13:25:16 DEBUG } 13:25:25 INFO 13:25:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:25:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:25:25 INFO [loop_until]: OK (rc = 0) 13:25:25 DEBUG --- stdout --- 13:25:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 233m 1% 3843Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 694m 2% 19052Mi 66% 13:25:25 DEBUG --- stderr --- 13:25:25 DEBUG 13:25:29 INFO 13:25:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:25:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:25:29 INFO [loop_until]: OK (rc = 0) 13:25:29 DEBUG --- stdout --- 13:25:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 730m 13477Mi rcs-645c86fdd6-j4rsk 154m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:25:29 DEBUG --- stderr --- 13:25:29 DEBUG 13:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910775" 13:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910775" 13:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910775" 13:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910775" 13:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910775" 13:26:15 INFO [http_cmd]: http status code OK 13:26:15 DEBUG --- status code --- 13:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:15 DEBUG --- http response --- 13:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:15 INFO [http_cmd]: http status code OK 13:26:15 DEBUG --- status code --- 13:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:15 DEBUG --- http response --- 13:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:15 INFO [http_cmd]: http status code OK 13:26:15 DEBUG --- status code --- 13:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:15 DEBUG --- http response --- 13:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:15 INFO [http_cmd]: http status code OK 13:26:15 DEBUG --- status code --- 13:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:15 DEBUG --- http response --- 13:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:15 ERROR [http_cmd]: ERROR 13:26:15 ERROR --- status code --- 13:26:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:26:15 ERROR --- http response --- 13:26:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:26:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910775" 13:26:15 INFO [http_cmd]: http status code OK 13:26:15 DEBUG --- status code --- 13:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:15 DEBUG --- http response --- 13:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910775" 13:26:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910775" 13:26:15 INFO [http_cmd]: http status code OK 13:26:15 DEBUG --- status code --- 13:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:15 DEBUG --- http response --- 13:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:15 INFO [http_cmd]: http status code OK 13:26:15 DEBUG --- status code --- 13:26:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:15 DEBUG --- http response --- 13:26:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG --- status code --- 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG --- http response --- 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 DEBUG --- http response --- 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG { 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG "data": { 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:26:16 DEBUG "result": [ 13:26:16 DEBUG { 13:26:16 DEBUG "metric": { 13:26:16 DEBUG "pod": "ds-idrepo-0" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG [...] 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "0" 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG } 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG { 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG "data": { 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG "result": [ 13:26:16 DEBUG { 13:26:16 DEBUG "metric": { 13:26:16 DEBUG "pod": "ds-idrepo-0" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG [...] 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "0.07321175694282153" 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG } 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910776" 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG { 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG "data": { 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG "result": [ 13:26:16 DEBUG { 13:26:16 DEBUG "metric": { 13:26:16 DEBUG "pod": "ds-idrepo-0" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG [...] 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "470.0481983015836" 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG } 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG { 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG "data": { 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG "result": [ 13:26:16 DEBUG { 13:26:16 DEBUG "metric": { 13:26:16 DEBUG "instance": "10.142.0.11:9100" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG [...] 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG "instance": "10.142.0.51:9100" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "0.00043333333333445503" 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG } 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910776" 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG { 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG "data": { 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG "result": [ 13:26:16 DEBUG { 13:26:16 DEBUG "metric": { 13:26:16 DEBUG "instance": "10.142.0.11:9100" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG [...] 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG "instance": "10.142.0.51:9100" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "2378690.8666666667" 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG } 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910776" 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG { 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG "data": { 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG "result": [ 13:26:16 DEBUG { 13:26:16 DEBUG "metric": { 13:26:16 DEBUG "instance": "10.142.0.11:9100" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG [...] 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG "instance": "10.142.0.51:9100" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "2395643.4666666673" 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG } 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910776" 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG { 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG "data": { 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG "result": [ 13:26:16 DEBUG { 13:26:16 DEBUG "metric": { 13:26:16 DEBUG "instance": "10.142.0.11:9100" 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG }, 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG "value": [ 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:26:16 DEBUG 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG [...] 13:26:16 DEBUG { 13:26:16 DEBUG 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG 13:26:16 DEBUG "data": { 13:26:16 DEBUG 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG "instance": "10.142.0.51:9100" 13:26:16 DEBUG "result": [ 13:26:16 DEBUG }, 13:26:16 DEBUG { 13:26:16 DEBUG "value": [ 13:26:16 DEBUG "metric": { 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:26:16 DEBUG "6690.133333333334" 13:26:16 DEBUG }, 13:26:16 DEBUG ] 13:26:16 DEBUG "value": [ 13:26:16 DEBUG } 13:26:16 DEBUG 13:26:16 DEBUG ] 13:26:16 DEBUG 13:26:16 DEBUG } 13:26:16 DEBUG 13:26:16 DEBUG } 13:26:16 DEBUG [...] 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "0.1057823236432017" 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG } 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910776" 13:26:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910776" 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG { 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG "data": { 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG "result": [ 13:26:16 DEBUG { 13:26:16 DEBUG "metric": { 13:26:16 DEBUG "instance": "10.142.0.11:9100" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG [...] 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG "instance": "10.142.0.51:9100" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "0" 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG } 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG { 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG "data": { 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG "result": [ 13:26:16 DEBUG { 13:26:16 DEBUG "metric": { 13:26:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 INFO [http_cmd]: http status code OK 13:26:16 DEBUG [...] 13:26:16 DEBUG --- status code --- 13:26:16 DEBUG 13:26:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:26:16 DEBUG 13:26:16 DEBUG --- http response --- 13:26:16 DEBUG 13:26:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:26:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:26:16 DEBUG ----- output ----- 13:26:16 DEBUG }, 13:26:16 DEBUG { 13:26:16 DEBUG "value": [ 13:26:16 DEBUG "status": "success", 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "data": { 13:26:16 DEBUG "14987952128" 13:26:16 DEBUG "resultType": "vector", 13:26:16 DEBUG ] 13:26:16 DEBUG "result": [ 13:26:16 DEBUG } 13:26:16 DEBUG { 13:26:16 DEBUG ] 13:26:16 DEBUG "metric": { 13:26:16 DEBUG } 13:26:16 DEBUG "pod": "ds-idrepo-0" 13:26:16 DEBUG } 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG [...] 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG 13:26:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:26:16 DEBUG }, 13:26:16 DEBUG "value": [ 13:26:16 DEBUG 1735910776, 13:26:16 DEBUG "758779904" 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG ] 13:26:16 DEBUG } 13:26:16 DEBUG } 13:26:25 INFO 13:26:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:26:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:26:25 INFO [loop_until]: OK (rc = 0) 13:26:25 DEBUG --- stdout --- 13:26:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 234m 1% 3843Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 140m 0% 19058Mi 67% 13:26:25 DEBUG --- stderr --- 13:26:25 DEBUG 13:26:29 INFO 13:26:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:26:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:26:29 INFO [loop_until]: OK (rc = 0) 13:26:29 DEBUG --- stdout --- 13:26:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 124m 13477Mi rcs-645c86fdd6-j4rsk 127m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:26:29 DEBUG --- stderr --- 13:26:29 DEBUG 13:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910835" 13:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910835" 13:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910835" 13:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910835" 13:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910835" 13:27:15 INFO [http_cmd]: http status code OK 13:27:15 DEBUG --- status code --- 13:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:15 DEBUG --- http response --- 13:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:15 INFO [http_cmd]: http status code OK 13:27:15 INFO [http_cmd]: http status code OK 13:27:15 ERROR [http_cmd]: ERROR 13:27:15 DEBUG --- status code --- 13:27:15 INFO [http_cmd]: http status code OK 13:27:15 DEBUG --- status code --- 13:27:15 ERROR --- status code --- 13:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:15 DEBUG --- status code --- 13:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:27:15 DEBUG --- http response --- 13:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:15 DEBUG --- http response --- 13:27:15 ERROR --- http response --- 13:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:15 DEBUG --- http response --- 13:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910835" 13:27:15 INFO [http_cmd]: http status code OK 13:27:15 DEBUG --- status code --- 13:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:15 DEBUG --- http response --- 13:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910835" 13:27:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910835" 13:27:15 INFO [http_cmd]: http status code OK 13:27:15 INFO [http_cmd]: http status code OK 13:27:15 DEBUG --- status code --- 13:27:15 DEBUG --- status code --- 13:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:15 DEBUG --- http response --- 13:27:15 DEBUG --- http response --- 13:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- http response --- 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG { 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG "data": { 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG "result": [ 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG { 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG "metric": { 13:27:16 DEBUG { 13:27:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG }, 13:27:16 DEBUG "data": { 13:27:16 DEBUG "value": [ 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG 13:27:16 DEBUG "result": [ 13:27:16 DEBUG 13:27:16 DEBUG { 13:27:16 DEBUG 13:27:16 DEBUG "metric": { 13:27:16 DEBUG [...] 13:27:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:27:16 DEBUG 13:27:16 DEBUG }, 13:27:16 DEBUG 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:27:16 DEBUG 13:27:16 DEBUG }, 13:27:16 DEBUG 13:27:16 DEBUG "value": [ 13:27:16 DEBUG [...] 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG 13:27:16 DEBUG "0" 13:27:16 DEBUG 13:27:16 DEBUG ] 13:27:16 DEBUG 13:27:16 DEBUG } 13:27:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:27:16 DEBUG ] 13:27:16 DEBUG }, 13:27:16 DEBUG } 13:27:16 DEBUG "value": [ 13:27:16 DEBUG } 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG "0.0002628500126634804" 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910836" 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG { 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG "data": { 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG "result": [ 13:27:16 DEBUG { 13:27:16 DEBUG "metric": { 13:27:16 DEBUG "instance": "10.142.0.11:9100" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG [...] 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG "instance": "10.142.0.51:9100" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG "0.001333333333333068" 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG { 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG "data": { 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG "result": [ 13:27:16 DEBUG { 13:27:16 DEBUG "metric": { 13:27:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG [...] 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG "0" 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910836" 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG { 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG "data": { 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG "result": [ 13:27:16 DEBUG { 13:27:16 DEBUG "metric": { 13:27:16 DEBUG "instance": "10.142.0.11:9100" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG [...] 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG "instance": "10.142.0.51:9100" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG "2025746.3666666667" 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910836" 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG { 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG "data": { 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG "result": [ 13:27:16 DEBUG { 13:27:16 DEBUG "metric": { 13:27:16 DEBUG "instance": "10.142.0.11:9100" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG [...] 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG "instance": "10.142.0.51:9100" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG "2044860.8333333337" 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910836" 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG { 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG "data": { 13:27:16 DEBUG { 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG "result": [ 13:27:16 DEBUG "data": { 13:27:16 DEBUG { 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG "metric": { 13:27:16 DEBUG "result": [ 13:27:16 DEBUG "instance": "10.142.0.11:9100" 13:27:16 DEBUG { 13:27:16 DEBUG }, 13:27:16 DEBUG "metric": { 13:27:16 DEBUG "value": [ 13:27:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:27:16 DEBUG 13:27:16 DEBUG }, 13:27:16 DEBUG 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG [...] 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG [...] 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG "instance": "10.142.0.51:9100" 13:27:16 DEBUG 13:27:16 DEBUG }, 13:27:16 DEBUG 13:27:16 DEBUG "value": [ 13:27:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG }, 13:27:16 DEBUG "29764.266666666666" 13:27:16 DEBUG "value": [ 13:27:16 DEBUG ] 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG } 13:27:16 DEBUG "0.03956104613177582" 13:27:16 DEBUG ] 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910836" 13:27:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910836" 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG { 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG "data": { 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG "result": [ 13:27:16 DEBUG { 13:27:16 DEBUG "metric": { 13:27:16 DEBUG "instance": "10.142.0.11:9100" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG [...] 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG "instance": "10.142.0.51:9100" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG "0.001000000000000038" 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG { 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG "data": { 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG "result": [ 13:27:16 DEBUG { 13:27:16 DEBUG "metric": { 13:27:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG [...] 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG "14987825152" 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:16 INFO [http_cmd]: http status code OK 13:27:16 DEBUG --- status code --- 13:27:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:27:16 DEBUG --- http response --- 13:27:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:27:16 DEBUG ----- output ----- 13:27:16 DEBUG { 13:27:16 DEBUG "status": "success", 13:27:16 DEBUG "data": { 13:27:16 DEBUG "resultType": "vector", 13:27:16 DEBUG "result": [ 13:27:16 DEBUG { 13:27:16 DEBUG "metric": { 13:27:16 DEBUG "pod": "ds-idrepo-0" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG [...] 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG 13:27:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:27:16 DEBUG }, 13:27:16 DEBUG "value": [ 13:27:16 DEBUG 1735910836, 13:27:16 DEBUG "758779904" 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG ] 13:27:16 DEBUG } 13:27:16 DEBUG } 13:27:25 INFO 13:27:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:27:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:27:25 INFO [loop_until]: OK (rc = 0) 13:27:25 DEBUG --- stdout --- 13:27:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 210m 1% 3865Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 196m 0% 19058Mi 67% 13:27:25 DEBUG --- stderr --- 13:27:25 DEBUG 13:27:29 INFO 13:27:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:27:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:27:29 INFO [loop_until]: OK (rc = 0) 13:27:29 DEBUG --- stdout --- 13:27:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 132m 13477Mi rcs-645c86fdd6-j4rsk 127m 823Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:27:29 DEBUG --- stderr --- 13:27:29 DEBUG 13:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910895" 13:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910895" 13:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910895" 13:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910895" 13:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910895" 13:28:15 INFO [http_cmd]: http status code OK 13:28:15 INFO [http_cmd]: http status code OK 13:28:15 INFO [http_cmd]: http status code OK 13:28:15 INFO [http_cmd]: http status code OK 13:28:15 DEBUG --- status code --- 13:28:15 DEBUG --- status code --- 13:28:15 DEBUG --- status code --- 13:28:15 DEBUG --- status code --- 13:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:15 DEBUG --- http response --- 13:28:15 DEBUG --- http response --- 13:28:15 DEBUG --- http response --- 13:28:15 DEBUG --- http response --- 13:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:15 ERROR [http_cmd]: ERROR 13:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:15 ERROR --- status code --- 13:28:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:28:15 ERROR --- http response --- 13:28:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:28:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910895" 13:28:15 INFO [http_cmd]: http status code OK 13:28:15 DEBUG --- status code --- 13:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:15 DEBUG --- http response --- 13:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910895" 13:28:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910895" 13:28:15 INFO [http_cmd]: http status code OK 13:28:15 DEBUG --- status code --- 13:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:15 DEBUG --- http response --- 13:28:15 INFO [http_cmd]: http status code OK 13:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:15 DEBUG --- status code --- 13:28:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:15 DEBUG --- http response --- 13:28:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- http response --- 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG { 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG "data": { 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "result": [ 13:28:16 DEBUG { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "pod": "ds-cts-0" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG [...] 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:28:16 DEBUG 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:28:16 DEBUG 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG 13:28:16 DEBUG { 13:28:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG }, 13:28:16 DEBUG "data": { 13:28:16 DEBUG "value": [ 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG "result": [ 13:28:16 DEBUG "0" 13:28:16 DEBUG { 13:28:16 DEBUG ] 13:28:16 DEBUG "metric": { 13:28:16 DEBUG } 13:28:16 DEBUG "pod": "ds-cts-0" 13:28:16 DEBUG ] 13:28:16 DEBUG }, 13:28:16 DEBUG } 13:28:16 DEBUG "value": [ 13:28:16 DEBUG } 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG [...] 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG "0" 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG { 13:28:16 DEBUG { 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG "data": { 13:28:16 DEBUG "data": { 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "result": [ 13:28:16 DEBUG "result": [ 13:28:16 DEBUG { 13:28:16 DEBUG { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "pod": "ds-cts-0" 13:28:16 DEBUG "instance": "10.142.0.11:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG [...] 13:28:16 DEBUG [...] 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:28:16 DEBUG "instance": "10.142.0.51:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG "515.5119249889874" 13:28:16 DEBUG "0.0006333333333344626" 13:28:16 DEBUG ] 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 DEBUG ] 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910896" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG { 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG "data": { 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "result": [ 13:28:16 DEBUG { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "instance": "10.142.0.11:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG [...] 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG "instance": "10.142.0.51:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG "715775.1000000002" 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910896" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG { 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG "data": { 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "result": [ 13:28:16 DEBUG { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "instance": "10.142.0.11:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG [...] 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG "instance": "10.142.0.51:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG "735923.1666666669" 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910896" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG { 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG "data": { 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "result": [ 13:28:16 DEBUG { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "instance": "10.142.0.11:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG [...] 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG "instance": "10.142.0.51:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG "24302.933333333334" 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG { 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG "data": { 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "result": [ 13:28:16 DEBUG { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG [...] 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG "0.042773339084847775" 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910896" 13:28:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910896" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG { 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG "data": { 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "result": [ 13:28:16 DEBUG { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "instance": "10.142.0.11:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG [...] 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG "instance": "10.142.0.51:9100" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG "0.000666666666666534" 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG { 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG "data": { 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "result": [ 13:28:16 DEBUG { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG [...] 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:28:16 INFO [http_cmd]: http status code OK 13:28:16 DEBUG }, 13:28:16 DEBUG --- status code --- 13:28:16 DEBUG "value": [ 13:28:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG --- http response --- 13:28:16 DEBUG "14985072640" 13:28:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:28:16 DEBUG ] 13:28:16 DEBUG ----- output ----- 13:28:16 DEBUG } 13:28:16 DEBUG { 13:28:16 DEBUG ] 13:28:16 DEBUG "status": "success", 13:28:16 DEBUG } 13:28:16 DEBUG "data": { 13:28:16 DEBUG } 13:28:16 DEBUG "resultType": "vector", 13:28:16 DEBUG "result": [ 13:28:16 DEBUG { 13:28:16 DEBUG "metric": { 13:28:16 DEBUG "pod": "ds-idrepo-0" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG [...] 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG 13:28:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:28:16 DEBUG }, 13:28:16 DEBUG "value": [ 13:28:16 DEBUG 1735910896, 13:28:16 DEBUG "758779904" 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG ] 13:28:16 DEBUG } 13:28:16 DEBUG } 13:28:25 INFO 13:28:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:28:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:28:25 INFO [loop_until]: OK (rc = 0) 13:28:25 DEBUG --- stdout --- 13:28:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 143m 0% 3868Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 107m 0% 19054Mi 66% 13:28:25 DEBUG --- stderr --- 13:28:25 DEBUG 13:28:29 INFO 13:28:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:28:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:28:29 INFO [loop_until]: OK (rc = 0) 13:28:29 DEBUG --- stdout --- 13:28:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 67m 13477Mi rcs-645c86fdd6-j4rsk 36m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:28:29 DEBUG --- stderr --- 13:28:29 DEBUG 13:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910955" 13:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910955" 13:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735910955" 13:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910955" 13:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910955" 13:29:15 INFO [http_cmd]: http status code OK 13:29:15 DEBUG --- status code --- 13:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:15 DEBUG --- http response --- 13:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:15 INFO [http_cmd]: http status code OK 13:29:15 DEBUG --- status code --- 13:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:15 DEBUG --- http response --- 13:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:15 ERROR [http_cmd]: ERROR 13:29:15 ERROR --- status code --- 13:29:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:29:15 ERROR --- http response --- 13:29:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:29:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:29:15 INFO [http_cmd]: http status code OK 13:29:15 DEBUG --- status code --- 13:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:15 DEBUG --- http response --- 13:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:15 INFO [http_cmd]: http status code OK 13:29:15 DEBUG --- status code --- 13:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:15 DEBUG --- http response --- 13:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910955" 13:29:15 INFO [http_cmd]: http status code OK 13:29:15 DEBUG --- status code --- 13:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:15 DEBUG --- http response --- 13:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910955" 13:29:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910955" 13:29:15 INFO [http_cmd]: http status code OK 13:29:15 DEBUG --- status code --- 13:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:15 DEBUG --- http response --- 13:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:15 INFO [http_cmd]: http status code OK 13:29:15 DEBUG --- status code --- 13:29:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:15 DEBUG --- http response --- 13:29:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG --- status code --- 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG --- status code --- 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 DEBUG --- status code --- 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735910956, "0" ] } ] } } 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG ----- output ----- 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG { 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG "status": "success", 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG "data": { 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG "resultType": "vector", 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG "result": [ 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:29:16 DEBUG { 13:29:16 DEBUG "metric": { 13:29:16 DEBUG "pod": "ds-cts-0" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG [...] 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 1735910956, 13:29:16 DEBUG "0" 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG } 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735910956, "437.5183689906871" ] } ] } } 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:29:16 DEBUG ----- output ----- 13:29:16 DEBUG { 13:29:16 DEBUG "status": "success", 13:29:16 DEBUG "data": { 13:29:16 DEBUG "resultType": "vector", 13:29:16 DEBUG "result": [ 13:29:16 DEBUG { 13:29:16 DEBUG "metric": { 13:29:16 DEBUG "instance": "10.142.0.11:9100" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG [...] 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG "instance": "10.142.0.51:9100" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 1735910956, 13:29:16 DEBUG "0.0006666666666622707" 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG } 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910956" 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:29:16 DEBUG ----- output ----- 13:29:16 DEBUG { 13:29:16 DEBUG "status": "success", 13:29:16 DEBUG "data": { 13:29:16 DEBUG "resultType": "vector", 13:29:16 DEBUG "result": [ 13:29:16 DEBUG { 13:29:16 DEBUG "metric": { 13:29:16 DEBUG "instance": "10.142.0.11:9100" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG [...] 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG "instance": "10.142.0.51:9100" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 1735910956, 13:29:16 DEBUG "75624.2" 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG } 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910956" 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:29:16 DEBUG ----- output ----- 13:29:16 DEBUG { 13:29:16 DEBUG "status": "success", 13:29:16 DEBUG "data": { 13:29:16 DEBUG "resultType": "vector", 13:29:16 DEBUG "result": [ 13:29:16 DEBUG { 13:29:16 DEBUG "metric": { 13:29:16 DEBUG "instance": "10.142.0.11:9100" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG [...] 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG "instance": "10.142.0.51:9100" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 1735910956, 13:29:16 DEBUG "95145.33333333331" 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG } 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735910956" 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:29:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:29:16 DEBUG ----- output ----- 13:29:16 DEBUG ----- output ----- 13:29:16 DEBUG { 13:29:16 DEBUG { 13:29:16 DEBUG "status": "success", 13:29:16 DEBUG "status": "success", 13:29:16 DEBUG "data": { 13:29:16 DEBUG "data": { 13:29:16 DEBUG "resultType": "vector", 13:29:16 DEBUG "resultType": "vector", 13:29:16 DEBUG "result": [ 13:29:16 DEBUG "result": [ 13:29:16 DEBUG { 13:29:16 DEBUG { 13:29:16 DEBUG "metric": { 13:29:16 DEBUG "metric": { 13:29:16 DEBUG "instance": "10.142.0.11:9100" 13:29:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:29:16 DEBUG }, 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG [...] 13:29:16 DEBUG [...] 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG "instance": "10.142.0.51:9100" 13:29:16 DEBUG }, 13:29:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:29:16 DEBUG "value": [ 13:29:16 DEBUG }, 13:29:16 DEBUG 1735910956, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG "22664.533333333333" 13:29:16 DEBUG 1735910956, 13:29:16 DEBUG ] 13:29:16 DEBUG "0.0374760599171861" 13:29:16 DEBUG } 13:29:16 DEBUG ] 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG } 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG } 13:29:16 DEBUG } 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735910956" 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735910956" 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:29:16 DEBUG ----- output ----- 13:29:16 DEBUG { 13:29:16 DEBUG "status": "success", 13:29:16 DEBUG "data": { 13:29:16 DEBUG "resultType": "vector", 13:29:16 DEBUG "result": [ 13:29:16 DEBUG { 13:29:16 DEBUG "metric": { 13:29:16 DEBUG "instance": "10.142.0.11:9100" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG [...] 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG "instance": "10.142.0.51:9100" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 1735910956, 13:29:16 DEBUG "0.00033333333333350387" 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG } 13:29:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735910956" 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG --- http response --- 13:29:16 INFO [http_cmd]: http status code OK 13:29:16 DEBUG --- status code --- 13:29:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:29:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:29:16 DEBUG ----- output ----- 13:29:16 DEBUG --- http response --- 13:29:16 DEBUG { 13:29:16 DEBUG "status": "success", 13:29:16 DEBUG "data": { 13:29:16 DEBUG "resultType": "vector", 13:29:16 DEBUG "result": [ 13:29:16 DEBUG { 13:29:16 DEBUG "metric": { 13:29:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG [...] 13:29:16 DEBUG 13:29:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:29:16 DEBUG 13:29:16 DEBUG ----- output ----- 13:29:16 DEBUG 13:29:16 DEBUG { 13:29:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:29:16 DEBUG "status": "success", 13:29:16 DEBUG }, 13:29:16 DEBUG "data": { 13:29:16 DEBUG "value": [ 13:29:16 DEBUG "resultType": "vector", 13:29:16 DEBUG 1735910956, 13:29:16 DEBUG "result": [ 13:29:16 DEBUG "14984691712" 13:29:16 DEBUG { 13:29:16 DEBUG ] 13:29:16 DEBUG "metric": { 13:29:16 DEBUG } 13:29:16 DEBUG "pod": "ds-idrepo-0" 13:29:16 DEBUG ] 13:29:16 DEBUG }, 13:29:16 DEBUG } 13:29:16 DEBUG "value": [ 13:29:16 DEBUG } 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG [...] 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG 13:29:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:29:16 DEBUG }, 13:29:16 DEBUG "value": [ 13:29:16 DEBUG 1735910956, 13:29:16 DEBUG "758788096" 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG ] 13:29:16 DEBUG } 13:29:16 DEBUG } 13:29:25 INFO 13:29:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:29:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:29:25 INFO [loop_until]: OK (rc = 0) 13:29:25 DEBUG --- stdout --- 13:29:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3847Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 85m 0% 19055Mi 67% 13:29:25 DEBUG --- stderr --- 13:29:25 DEBUG 13:29:29 INFO 13:29:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:29:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:29:29 INFO [loop_until]: OK (rc = 0) 13:29:29 DEBUG --- stdout --- 13:29:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:29:29 DEBUG --- stderr --- 13:29:29 DEBUG 13:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911015" 13:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911015" 13:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911015" 13:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911015" 13:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911015" 13:30:15 INFO [http_cmd]: http status code OK 13:30:15 DEBUG --- status code --- 13:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:15 DEBUG --- http response --- 13:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:15 ERROR [http_cmd]: ERROR 13:30:15 ERROR --- status code --- 13:30:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:30:15 ERROR --- http response --- 13:30:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:30:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:30:15 INFO [http_cmd]: http status code OK 13:30:15 DEBUG --- status code --- 13:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:15 DEBUG --- http response --- 13:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:15 INFO [http_cmd]: http status code OK 13:30:15 INFO [http_cmd]: http status code OK 13:30:15 DEBUG --- status code --- 13:30:15 DEBUG --- status code --- 13:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:15 DEBUG --- http response --- 13:30:15 DEBUG --- http response --- 13:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911015" 13:30:15 INFO [http_cmd]: http status code OK 13:30:15 DEBUG --- status code --- 13:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:15 DEBUG --- http response --- 13:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911015" 13:30:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911015" 13:30:15 INFO [http_cmd]: http status code OK 13:30:15 DEBUG --- status code --- 13:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:15 DEBUG --- http response --- 13:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:15 INFO [http_cmd]: http status code OK 13:30:15 DEBUG --- status code --- 13:30:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:15 DEBUG --- http response --- 13:30:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 DEBUG --- status code --- 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG { 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG "data": { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "result": [ 13:30:16 DEBUG { 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG "pod": "ds-idrepo-0" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG "0" 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:30:16 DEBUG ----- output ----- 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG { 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG "data": { 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 DEBUG "result": [ 13:30:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:30:16 DEBUG { 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG "pod": "ds-cts-0" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG "0" 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG { 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG "data": { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "result": [ 13:30:16 DEBUG { 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG "pod": "ds-idrepo-0" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG "645.6711987990564" 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG { 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG "data": { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "result": [ 13:30:16 DEBUG { 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "instance": "10.142.0.11:9100" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG "instance": "10.142.0.51:9100" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG "0.00043333333333445503" 13:30:16 DEBUG ] 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911016" 13:30:16 DEBUG } 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG { 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG "data": { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "result": [ 13:30:16 DEBUG { 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "instance": "10.142.0.11:9100" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG "instance": "10.142.0.51:9100" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG "76138.03333333334" 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG { 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG "data": { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "result": [ 13:30:16 DEBUG { 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "instance": "10.142.0.11:9100" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG "instance": "10.142.0.51:9100" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG "95897.86666666668" 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911016" 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG "data": { 13:30:16 DEBUG { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG "result": [ 13:30:16 DEBUG "data": { 13:30:16 DEBUG { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "result": [ 13:30:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:30:16 DEBUG { 13:30:16 DEBUG }, 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "value": [ 13:30:16 DEBUG "instance": "10.142.0.11:9100" 13:30:16 DEBUG 13:30:16 DEBUG }, 13:30:16 DEBUG 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:30:16 DEBUG 13:30:16 DEBUG }, 13:30:16 DEBUG 13:30:16 DEBUG "value": [ 13:30:16 DEBUG "instance": "10.142.0.51:9100" 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG }, 13:30:16 DEBUG "0.03838630896016904" 13:30:16 DEBUG "value": [ 13:30:16 DEBUG ] 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG } 13:30:16 DEBUG "18158.93333333333" 13:30:16 DEBUG ] 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911016" 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG { 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG "data": { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "result": [ 13:30:16 DEBUG { 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "instance": "10.142.0.11:9100" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG "instance": "10.142.0.51:9100" 13:30:16 DEBUG }, 13:30:16 DEBUG "value": [ 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911016" 13:30:16 DEBUG 1735911016, 13:30:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911016" 13:30:16 DEBUG "0" 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 INFO [http_cmd]: http status code OK 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG --- status code --- 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:30:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG --- http response --- 13:30:16 DEBUG { 13:30:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG ----- output ----- 13:30:16 DEBUG "data": { 13:30:16 DEBUG { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "status": "success", 13:30:16 DEBUG "result": [ 13:30:16 DEBUG "data": { 13:30:16 DEBUG { 13:30:16 DEBUG "resultType": "vector", 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "result": [ 13:30:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:30:16 DEBUG { 13:30:16 DEBUG }, 13:30:16 DEBUG "metric": { 13:30:16 DEBUG "value": [ 13:30:16 DEBUG "pod": "ds-idrepo-0" 13:30:16 DEBUG 13:30:16 DEBUG }, 13:30:16 DEBUG 13:30:16 DEBUG "value": [ 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG [...] 13:30:16 DEBUG 13:30:16 DEBUG 13:30:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:30:16 DEBUG 13:30:16 DEBUG }, 13:30:16 DEBUG 13:30:16 DEBUG "value": [ 13:30:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG }, 13:30:16 DEBUG "14987554816" 13:30:16 DEBUG "value": [ 13:30:16 DEBUG ] 13:30:16 DEBUG 1735911016, 13:30:16 DEBUG } 13:30:16 DEBUG "758779904" 13:30:16 DEBUG ] 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:16 DEBUG ] 13:30:16 DEBUG } 13:30:16 DEBUG } 13:30:25 INFO 13:30:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:30:25 INFO [loop_until]: OK (rc = 0) 13:30:25 DEBUG --- stdout --- 13:30:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 112m 0% 3858Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 90m 0% 19056Mi 67% 13:30:25 DEBUG --- stderr --- 13:30:25 DEBUG 13:30:29 INFO 13:30:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:30:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:30:29 INFO [loop_until]: OK (rc = 0) 13:30:29 DEBUG --- stdout --- 13:30:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:30:29 DEBUG --- stderr --- 13:30:29 DEBUG 13:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911075" 13:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911075" 13:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911075" 13:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911075" 13:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911075" 13:31:15 INFO [http_cmd]: http status code OK 13:31:15 DEBUG --- status code --- 13:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:15 DEBUG --- http response --- 13:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:15 INFO [http_cmd]: http status code OK 13:31:15 DEBUG --- status code --- 13:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:15 DEBUG --- http response --- 13:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:15 INFO [http_cmd]: http status code OK 13:31:15 DEBUG --- status code --- 13:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:15 DEBUG --- http response --- 13:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:15 ERROR [http_cmd]: ERROR 13:31:15 INFO [http_cmd]: http status code OK 13:31:15 ERROR --- status code --- 13:31:15 DEBUG --- status code --- 13:31:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:31:15 ERROR --- http response --- 13:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:31:15 DEBUG --- http response --- 13:31:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911075" 13:31:15 INFO [http_cmd]: http status code OK 13:31:15 DEBUG --- status code --- 13:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:15 DEBUG --- http response --- 13:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911075" 13:31:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911075" 13:31:15 INFO [http_cmd]: http status code OK 13:31:15 DEBUG --- status code --- 13:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:15 DEBUG --- http response --- 13:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:15 INFO [http_cmd]: http status code OK 13:31:15 DEBUG --- status code --- 13:31:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:15 DEBUG --- http response --- 13:31:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- status code --- 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- status code --- 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG --- http response --- 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG { 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG "data": { 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG "result": [ 13:31:16 DEBUG { 13:31:16 DEBUG "metric": { 13:31:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG [...] 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG "pod": "ds-cts-0" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG "0" 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG { 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG "data": { 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG "result": [ 13:31:16 DEBUG { 13:31:16 DEBUG "metric": { 13:31:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG [...] 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG "pod": "ds-cts-0" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG "0.0017037721935781267" 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG { 13:31:16 DEBUG --- http response --- 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911076" 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:31:16 DEBUG "data": { 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG { 13:31:16 DEBUG "result": [ 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG { 13:31:16 DEBUG "data": { 13:31:16 DEBUG "metric": { 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:31:16 DEBUG "result": [ 13:31:16 DEBUG }, 13:31:16 DEBUG { 13:31:16 DEBUG "value": [ 13:31:16 DEBUG "metric": { 13:31:16 DEBUG 13:31:16 DEBUG "instance": "10.142.0.11:9100" 13:31:16 DEBUG 13:31:16 DEBUG }, 13:31:16 DEBUG 13:31:16 DEBUG "value": [ 13:31:16 DEBUG [...] 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG [...] 13:31:16 DEBUG "pod": "ds-cts-0" 13:31:16 DEBUG 13:31:16 DEBUG }, 13:31:16 DEBUG 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG "instance": "10.142.0.51:9100" 13:31:16 DEBUG "455.46605566571196" 13:31:16 DEBUG }, 13:31:16 DEBUG ] 13:31:16 DEBUG "value": [ 13:31:16 DEBUG } 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG ] 13:31:16 DEBUG "0.00040000000000948907" 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG { 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG "data": { 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG "result": [ 13:31:16 DEBUG { 13:31:16 DEBUG "metric": { 13:31:16 DEBUG "instance": "10.142.0.11:9100" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG [...] 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG "instance": "10.142.0.51:9100" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG "74948.43333333333" 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG { 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG "data": { 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG "result": [ 13:31:16 DEBUG { 13:31:16 DEBUG "metric": { 13:31:16 DEBUG "instance": "10.142.0.11:9100" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG [...] 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG "instance": "10.142.0.51:9100" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG "94256.43333333332" 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG { 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG "data": { 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG "result": [ 13:31:16 DEBUG { 13:31:16 DEBUG "metric": { 13:31:16 DEBUG "instance": "10.142.0.11:9100" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG [...] 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG "instance": "10.142.0.51:9100" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG "19933.866666666665" 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG { 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG "data": { 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG "result": [ 13:31:16 DEBUG { 13:31:16 DEBUG "metric": { 13:31:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG [...] 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG "0.05084395977412004" 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG { 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG "data": { 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG "result": [ 13:31:16 DEBUG { 13:31:16 DEBUG "metric": { 13:31:16 DEBUG "instance": "10.142.0.11:9100" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG [...] 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG "instance": "10.142.0.51:9100" 13:31:16 DEBUG }, 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG "0" 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911076" 13:31:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911076" 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG { 13:31:16 INFO [http_cmd]: http status code OK 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG --- status code --- 13:31:16 DEBUG "data": { 13:31:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG --- http response --- 13:31:16 DEBUG "result": [ 13:31:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:31:16 DEBUG { 13:31:16 DEBUG ----- output ----- 13:31:16 DEBUG "metric": { 13:31:16 DEBUG { 13:31:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:31:16 DEBUG "status": "success", 13:31:16 DEBUG }, 13:31:16 DEBUG "data": { 13:31:16 DEBUG "value": [ 13:31:16 DEBUG "resultType": "vector", 13:31:16 DEBUG 13:31:16 DEBUG "result": [ 13:31:16 DEBUG 13:31:16 DEBUG { 13:31:16 DEBUG 13:31:16 DEBUG "metric": { 13:31:16 DEBUG [...] 13:31:16 DEBUG "pod": "ds-idrepo-0" 13:31:16 DEBUG 13:31:16 DEBUG }, 13:31:16 DEBUG 13:31:16 DEBUG "value": [ 13:31:16 DEBUG 13:31:16 DEBUG 13:31:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:31:16 DEBUG 13:31:16 DEBUG }, 13:31:16 DEBUG 13:31:16 DEBUG "value": [ 13:31:16 DEBUG [...] 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG 13:31:16 DEBUG "15004127232" 13:31:16 DEBUG 13:31:16 DEBUG ] 13:31:16 DEBUG 13:31:16 DEBUG } 13:31:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:31:16 DEBUG ] 13:31:16 DEBUG }, 13:31:16 DEBUG } 13:31:16 DEBUG "value": [ 13:31:16 DEBUG } 13:31:16 DEBUG 1735911076, 13:31:16 DEBUG "758779904" 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG ] 13:31:16 DEBUG } 13:31:16 DEBUG } 13:31:25 INFO 13:31:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:31:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:31:25 INFO [loop_until]: OK (rc = 0) 13:31:25 DEBUG --- stdout --- 13:31:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 115m 0% 3859Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 91m 0% 19054Mi 66% 13:31:25 DEBUG --- stderr --- 13:31:25 DEBUG 13:31:29 INFO 13:31:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:31:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:31:29 INFO [loop_until]: OK (rc = 0) 13:31:29 DEBUG --- stdout --- 13:31:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 2m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:31:29 DEBUG --- stderr --- 13:31:29 DEBUG 13:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911135" 13:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911135" 13:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911135" 13:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911135" 13:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911135" 13:32:15 INFO [http_cmd]: http status code OK 13:32:15 DEBUG --- status code --- 13:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:15 DEBUG --- http response --- 13:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:15 ERROR [http_cmd]: ERROR 13:32:15 ERROR --- status code --- 13:32:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:32:15 ERROR --- http response --- 13:32:15 INFO [http_cmd]: http status code OK 13:32:15 DEBUG --- status code --- 13:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:15 DEBUG --- http response --- 13:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:32:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:32:15 INFO [http_cmd]: http status code OK 13:32:15 DEBUG --- status code --- 13:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:15 DEBUG --- http response --- 13:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:15 INFO [http_cmd]: http status code OK 13:32:15 DEBUG --- status code --- 13:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:15 DEBUG --- http response --- 13:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911135" 13:32:15 INFO [http_cmd]: http status code OK 13:32:15 DEBUG --- status code --- 13:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:15 DEBUG --- http response --- 13:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911135" 13:32:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911135" 13:32:15 INFO [http_cmd]: http status code OK 13:32:15 DEBUG --- status code --- 13:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:15 DEBUG --- http response --- 13:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:15 INFO [http_cmd]: http status code OK 13:32:15 DEBUG --- status code --- 13:32:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:15 DEBUG --- http response --- 13:32:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911136" 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG { 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "data": { 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG "result": [ 13:32:16 DEBUG { 13:32:16 DEBUG "metric": { 13:32:16 DEBUG "pod": "ds-cts-0" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG [...] 13:32:16 DEBUG 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG }, 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:32:16 DEBUG "value": [ 13:32:16 DEBUG { 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "0" 13:32:16 DEBUG "data": { 13:32:16 DEBUG ] 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG "resultType": "vector", 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG } 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG "result": [ 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG ] 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG { 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG } 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG "metric": { 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG } 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG "pod": "ds-cts-0" 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG }, 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG { 13:32:16 DEBUG "value": [ 13:32:16 DEBUG { 13:32:16 DEBUG { 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "data": { 13:32:16 DEBUG 13:32:16 DEBUG "data": { 13:32:16 DEBUG "data": { 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG "result": [ 13:32:16 DEBUG [...] 13:32:16 DEBUG "result": [ 13:32:16 DEBUG "result": [ 13:32:16 DEBUG { 13:32:16 DEBUG 13:32:16 DEBUG { 13:32:16 DEBUG { 13:32:16 DEBUG "metric": { 13:32:16 DEBUG 13:32:16 DEBUG "metric": { 13:32:16 DEBUG "metric": { 13:32:16 DEBUG "instance": "10.142.0.11:9100" 13:32:16 DEBUG 13:32:16 DEBUG "pod": "ds-cts-0" 13:32:16 DEBUG "instance": "10.142.0.11:9100" 13:32:16 DEBUG }, 13:32:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:32:16 DEBUG }, 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG "0.000494131354292776" 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG [...] 13:32:16 DEBUG ] 13:32:16 DEBUG [...] 13:32:16 DEBUG [...] 13:32:16 DEBUG 13:32:16 DEBUG } 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG ] 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG } 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG "instance": "10.142.0.51:9100" 13:32:16 DEBUG } 13:32:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:32:16 DEBUG "instance": "10.142.0.51:9100" 13:32:16 DEBUG }, 13:32:16 DEBUG }, 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG "value": [ 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG "0.0011666666666627634" 13:32:16 DEBUG "541.0216592191749" 13:32:16 DEBUG "74180.13333333333" 13:32:16 DEBUG ] 13:32:16 DEBUG ] 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 DEBUG ] 13:32:16 DEBUG ] 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911136" 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG { 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "data": { 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG "result": [ 13:32:16 DEBUG { 13:32:16 DEBUG "metric": { 13:32:16 DEBUG "instance": "10.142.0.11:9100" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG [...] 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG "instance": "10.142.0.51:9100" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG "93424.56666666668" 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911136" 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG { 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "data": { 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG "result": [ 13:32:16 DEBUG { 13:32:16 DEBUG "metric": { 13:32:16 DEBUG "instance": "10.142.0.11:9100" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG [...] 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG "instance": "10.142.0.51:9100" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG "13926.400000000001" 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG { 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "data": { 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG "result": [ 13:32:16 DEBUG { 13:32:16 DEBUG "metric": { 13:32:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG [...] 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG "0.04551226318771246" 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911136" 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG { 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "data": { 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG "result": [ 13:32:16 DEBUG { 13:32:16 DEBUG "metric": { 13:32:16 DEBUG "instance": "10.142.0.11:9100" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG [...] 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG "instance": "10.142.0.51:9100" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG "0.001000000000000038" 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911136" 13:32:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911136" 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG --- http response --- 13:32:16 INFO [http_cmd]: http status code OK 13:32:16 DEBUG --- status code --- 13:32:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:32:16 DEBUG --- http response --- 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG { 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "data": { 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG "result": [ 13:32:16 DEBUG { 13:32:16 DEBUG "metric": { 13:32:16 DEBUG "pod": "ds-idrepo-0" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG [...] 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG "758779904" 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:32:16 DEBUG ----- output ----- 13:32:16 DEBUG { 13:32:16 DEBUG "status": "success", 13:32:16 DEBUG "data": { 13:32:16 DEBUG "resultType": "vector", 13:32:16 DEBUG "result": [ 13:32:16 DEBUG { 13:32:16 DEBUG "metric": { 13:32:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG [...] 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG 13:32:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:32:16 DEBUG }, 13:32:16 DEBUG "value": [ 13:32:16 DEBUG 1735911136, 13:32:16 DEBUG "14985740288" 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG ] 13:32:16 DEBUG } 13:32:16 DEBUG } 13:32:25 INFO 13:32:25 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:32:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:32:26 INFO [loop_until]: OK (rc = 0) 13:32:26 DEBUG --- stdout --- 13:32:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 113m 0% 3861Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 87m 0% 19055Mi 67% 13:32:26 DEBUG --- stderr --- 13:32:26 DEBUG 13:32:29 INFO 13:32:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:32:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:32:29 INFO [loop_until]: OK (rc = 0) 13:32:29 DEBUG --- stdout --- 13:32:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13476Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:32:29 DEBUG --- stderr --- 13:32:29 DEBUG 13:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911195" 13:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911195" 13:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911195" 13:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911195" 13:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911195" 13:33:15 INFO [http_cmd]: http status code OK 13:33:15 ERROR [http_cmd]: ERROR 13:33:15 INFO [http_cmd]: http status code OK 13:33:15 INFO [http_cmd]: http status code OK 13:33:15 DEBUG --- status code --- 13:33:15 INFO [http_cmd]: http status code OK 13:33:15 ERROR --- status code --- 13:33:15 DEBUG --- status code --- 13:33:15 DEBUG --- status code --- 13:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:15 DEBUG --- status code --- 13:33:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:15 DEBUG --- http response --- 13:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:15 ERROR --- http response --- 13:33:15 DEBUG --- http response --- 13:33:15 DEBUG --- http response --- 13:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:15 DEBUG --- http response --- 13:33:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911195" 13:33:15 INFO [http_cmd]: http status code OK 13:33:15 DEBUG --- status code --- 13:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:15 DEBUG --- http response --- 13:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911195" 13:33:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911195" 13:33:15 INFO [http_cmd]: http status code OK 13:33:15 DEBUG --- status code --- 13:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:15 DEBUG --- http response --- 13:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:15 INFO [http_cmd]: http status code OK 13:33:15 DEBUG --- status code --- 13:33:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:15 DEBUG --- http response --- 13:33:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-j4rsk" }, "value": [ 1735911196, "0" ] } ] } } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:33:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:33:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:33:16 DEBUG ----- output ----- 13:33:16 DEBUG ----- output ----- 13:33:16 DEBUG ----- output ----- 13:33:16 DEBUG { 13:33:16 DEBUG { 13:33:16 DEBUG { 13:33:16 DEBUG "status": "success", 13:33:16 DEBUG "status": "success", 13:33:16 DEBUG "status": "success", 13:33:16 DEBUG "data": { 13:33:16 DEBUG "data": { 13:33:16 DEBUG "data": { 13:33:16 DEBUG "resultType": "vector", 13:33:16 DEBUG "resultType": "vector", 13:33:16 DEBUG "resultType": "vector", 13:33:16 DEBUG "result": [ 13:33:16 DEBUG "result": [ 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG "result": [ 13:33:16 DEBUG { 13:33:16 DEBUG { 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG { 13:33:16 DEBUG "metric": { 13:33:16 DEBUG "metric": { 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG "metric": { 13:33:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:33:16 DEBUG "instance": "10.142.0.11:9100" 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG "instance": "10.142.0.11:9100" 13:33:16 DEBUG }, 13:33:16 DEBUG }, 13:33:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-j4rsk" }, "value": [ 1735911196, "253.43398094295262" ] } ] } } 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG "value": [ 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG [...] 13:33:16 DEBUG [...] 13:33:16 DEBUG [...] 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:33:16 DEBUG "instance": "10.142.0.51:9100" 13:33:16 DEBUG "instance": "10.142.0.51:9100" 13:33:16 DEBUG }, 13:33:16 DEBUG }, 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG "value": [ 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 1735911196, 13:33:16 DEBUG 1735911196, 13:33:16 DEBUG 1735911196, 13:33:16 DEBUG "0" 13:33:16 DEBUG "75689.6" 13:33:16 DEBUG "0.0005333333333406168" 13:33:16 DEBUG ] 13:33:16 DEBUG ] 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 DEBUG ] 13:33:16 DEBUG ] 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:33:16 DEBUG ----- output ----- 13:33:16 DEBUG { 13:33:16 DEBUG "status": "success", 13:33:16 DEBUG "data": { 13:33:16 DEBUG "resultType": "vector", 13:33:16 DEBUG "result": [ 13:33:16 DEBUG { 13:33:16 DEBUG "metric": { 13:33:16 DEBUG "instance": "10.142.0.11:9100" 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG [...] 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG "instance": "10.142.0.51:9100" 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 1735911196, 13:33:16 DEBUG "95357.53333333335" 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:33:16 DEBUG ----- output ----- 13:33:16 DEBUG { 13:33:16 DEBUG "status": "success", 13:33:16 DEBUG "data": { 13:33:16 DEBUG "resultType": "vector", 13:33:16 DEBUG "result": [ 13:33:16 DEBUG { 13:33:16 DEBUG "metric": { 13:33:16 DEBUG "instance": "10.142.0.11:9100" 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG [...] 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG "instance": "10.142.0.51:9100" 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 1735911196, 13:33:16 DEBUG "24849.066666666666" 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:33:16 DEBUG ----- output ----- 13:33:16 DEBUG { 13:33:16 DEBUG "status": "success", 13:33:16 DEBUG "data": { 13:33:16 DEBUG "resultType": "vector", 13:33:16 DEBUG "result": [ 13:33:16 DEBUG { 13:33:16 DEBUG "metric": { 13:33:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG [...] 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 1735911196, 13:33:16 DEBUG "0.042800968566330914" 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:33:16 DEBUG ----- output ----- 13:33:16 DEBUG { 13:33:16 DEBUG "status": "success", 13:33:16 DEBUG "data": { 13:33:16 DEBUG "resultType": "vector", 13:33:16 DEBUG "result": [ 13:33:16 DEBUG { 13:33:16 DEBUG "metric": { 13:33:16 DEBUG "instance": "10.142.0.11:9100" 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG [...] 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG "instance": "10.142.0.51:9100" 13:33:16 DEBUG }, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 1735911196, 13:33:16 DEBUG "0.0006666666666689025" 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911196" 13:33:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911196" 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG --- http response --- 13:33:16 INFO [http_cmd]: http status code OK 13:33:16 DEBUG --- status code --- 13:33:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:33:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:33:16 DEBUG --- http response --- 13:33:16 DEBUG ----- output ----- 13:33:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:33:16 DEBUG { 13:33:16 DEBUG ----- output ----- 13:33:16 DEBUG "status": "success", 13:33:16 DEBUG { 13:33:16 DEBUG "data": { 13:33:16 DEBUG "status": "success", 13:33:16 DEBUG "resultType": "vector", 13:33:16 DEBUG "data": { 13:33:16 DEBUG "result": [ 13:33:16 DEBUG "resultType": "vector", 13:33:16 DEBUG { 13:33:16 DEBUG "result": [ 13:33:16 DEBUG "metric": { 13:33:16 DEBUG { 13:33:16 DEBUG "pod": "ds-idrepo-0" 13:33:16 DEBUG "metric": { 13:33:16 DEBUG }, 13:33:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:33:16 DEBUG "value": [ 13:33:16 DEBUG }, 13:33:16 DEBUG 13:33:16 DEBUG "value": [ 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG [...] 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG [...] 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG 13:33:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:33:16 DEBUG 13:33:16 DEBUG }, 13:33:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:33:16 DEBUG "value": [ 13:33:16 DEBUG }, 13:33:16 DEBUG 1735911196, 13:33:16 DEBUG "value": [ 13:33:16 DEBUG "758779904" 13:33:16 DEBUG 1735911196, 13:33:16 DEBUG ] 13:33:16 DEBUG "14987730944" 13:33:16 DEBUG } 13:33:16 DEBUG ] 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 DEBUG ] 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:16 DEBUG } 13:33:26 INFO 13:33:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:33:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:33:26 INFO [loop_until]: OK (rc = 0) 13:33:26 DEBUG --- stdout --- 13:33:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 113m 0% 3863Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 84m 0% 19056Mi 67% 13:33:26 DEBUG --- stderr --- 13:33:26 DEBUG 13:33:29 INFO 13:33:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:33:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:33:29 INFO [loop_until]: OK (rc = 0) 13:33:29 DEBUG --- stdout --- 13:33:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:33:29 DEBUG --- stderr --- 13:33:29 DEBUG 13:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911255" 13:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911255" 13:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911255" 13:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911255" 13:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911255" 13:34:15 INFO [http_cmd]: http status code OK 13:34:15 DEBUG --- status code --- 13:34:15 INFO [http_cmd]: http status code OK 13:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:15 DEBUG --- status code --- 13:34:15 DEBUG --- http response --- 13:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:15 DEBUG --- http response --- 13:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:15 INFO [http_cmd]: http status code OK 13:34:15 DEBUG --- status code --- 13:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:15 DEBUG --- http response --- 13:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:15 ERROR [http_cmd]: ERROR 13:34:15 INFO [http_cmd]: http status code OK 13:34:15 ERROR --- status code --- 13:34:15 DEBUG --- status code --- 13:34:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:15 ERROR --- http response --- 13:34:15 DEBUG --- http response --- 13:34:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911255" 13:34:15 INFO [http_cmd]: http status code OK 13:34:15 DEBUG --- status code --- 13:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:15 DEBUG --- http response --- 13:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911255" 13:34:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911255" 13:34:15 INFO [http_cmd]: http status code OK 13:34:15 DEBUG --- status code --- 13:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:15 DEBUG --- http response --- 13:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:15 INFO [http_cmd]: http status code OK 13:34:15 DEBUG --- status code --- 13:34:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:15 DEBUG --- http response --- 13:34:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911256" 13:34:16 DEBUG --- status code --- 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911256" 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG { 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG "data": { 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG "result": [ 13:34:16 DEBUG { 13:34:16 DEBUG "metric": { 13:34:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG [...] 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG "pod": "ds-cts-0" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG "0" 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG { 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG { 13:34:16 DEBUG { 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG "data": { 13:34:16 DEBUG "data": { 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG "data": { 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG { 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG "result": [ 13:34:16 DEBUG "result": [ 13:34:16 DEBUG { 13:34:16 DEBUG "result": [ 13:34:16 DEBUG { 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG "metric": { 13:34:16 DEBUG { 13:34:16 DEBUG "metric": { 13:34:16 DEBUG "data": { 13:34:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:34:16 DEBUG "metric": { 13:34:16 DEBUG "instance": "10.142.0.11:9100" 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG }, 13:34:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:34:16 DEBUG }, 13:34:16 DEBUG "result": [ 13:34:16 DEBUG "value": [ 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG { 13:34:16 DEBUG 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 13:34:16 DEBUG "metric": { 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG "instance": "10.142.0.11:9100" 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG }, 13:34:16 DEBUG [...] 13:34:16 DEBUG 13:34:16 DEBUG [...] 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 13:34:16 DEBUG [...] 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG "pod": "ds-cts-0" 13:34:16 DEBUG 13:34:16 DEBUG "instance": "10.142.0.51:9100" 13:34:16 DEBUG [...] 13:34:16 DEBUG }, 13:34:16 DEBUG "pod": "ds-cts-0" 13:34:16 DEBUG }, 13:34:16 DEBUG 13:34:16 DEBUG "value": [ 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 13:34:16 DEBUG "0.0012427250816114262" 13:34:16 DEBUG "486.68151202749146" 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG 13:34:16 DEBUG ] 13:34:16 DEBUG ] 13:34:16 DEBUG "76767.30000000002" 13:34:16 DEBUG "instance": "10.142.0.51:9100" 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG "0.0004666666666632106" 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911256" 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG { 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG "data": { 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG "result": [ 13:34:16 DEBUG { 13:34:16 DEBUG "metric": { 13:34:16 DEBUG "instance": "10.142.0.11:9100" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG [...] 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG "instance": "10.142.0.51:9100" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG "95782.3" 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911256" 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG { 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG "data": { 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG "result": [ 13:34:16 DEBUG { 13:34:16 DEBUG "metric": { 13:34:16 DEBUG "instance": "10.142.0.11:9100" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG [...] 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG "instance": "10.142.0.51:9100" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG "19660.8" 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG { 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG "data": { 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG "result": [ 13:34:16 DEBUG { 13:34:16 DEBUG "metric": { 13:34:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG [...] 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG "0.044749282899144774" 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911256" 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG { 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG "data": { 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG "result": [ 13:34:16 DEBUG { 13:34:16 DEBUG "metric": { 13:34:16 DEBUG "instance": "10.142.0.11:9100" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG [...] 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG "instance": "10.142.0.51:9100" 13:34:16 DEBUG }, 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG "0.0006666666666674814" 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911256" 13:34:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911256" 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG --- http response --- 13:34:16 INFO [http_cmd]: http status code OK 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:34:16 DEBUG --- status code --- 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:34:16 DEBUG { 13:34:16 DEBUG --- http response --- 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:34:16 DEBUG "data": { 13:34:16 DEBUG ----- output ----- 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG { 13:34:16 DEBUG "result": [ 13:34:16 DEBUG "status": "success", 13:34:16 DEBUG { 13:34:16 DEBUG "data": { 13:34:16 DEBUG "metric": { 13:34:16 DEBUG "resultType": "vector", 13:34:16 DEBUG "pod": "ds-idrepo-0" 13:34:16 DEBUG "result": [ 13:34:16 DEBUG }, 13:34:16 DEBUG { 13:34:16 DEBUG "value": [ 13:34:16 DEBUG "metric": { 13:34:16 DEBUG 13:34:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:34:16 DEBUG 13:34:16 DEBUG }, 13:34:16 DEBUG 13:34:16 DEBUG "value": [ 13:34:16 DEBUG [...] 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG 13:34:16 DEBUG [...] 13:34:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:34:16 DEBUG 13:34:16 DEBUG }, 13:34:16 DEBUG 13:34:16 DEBUG "value": [ 13:34:16 DEBUG 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:34:16 DEBUG "758779904" 13:34:16 DEBUG }, 13:34:16 DEBUG ] 13:34:16 DEBUG "value": [ 13:34:16 DEBUG } 13:34:16 DEBUG 1735911256, 13:34:16 DEBUG ] 13:34:16 DEBUG "14988836864" 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:16 DEBUG ] 13:34:16 DEBUG } 13:34:16 DEBUG } 13:34:26 INFO 13:34:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:34:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:34:26 INFO [loop_until]: OK (rc = 0) 13:34:26 DEBUG --- stdout --- 13:34:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 116m 0% 3835Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 89m 0% 19053Mi 66% 13:34:26 DEBUG --- stderr --- 13:34:26 DEBUG 13:34:29 INFO 13:34:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:34:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:34:29 INFO [loop_until]: OK (rc = 0) 13:34:29 DEBUG --- stdout --- 13:34:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:34:29 DEBUG --- stderr --- 13:34:29 DEBUG 13:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911315" 13:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911315" 13:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911315" 13:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911315" 13:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911315" 13:35:15 INFO [http_cmd]: http status code OK 13:35:15 DEBUG --- status code --- 13:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:15 DEBUG --- http response --- 13:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:15 INFO [http_cmd]: http status code OK 13:35:15 DEBUG --- status code --- 13:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:15 DEBUG --- http response --- 13:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:15 INFO [http_cmd]: http status code OK 13:35:15 DEBUG --- status code --- 13:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:15 DEBUG --- http response --- 13:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:15 ERROR [http_cmd]: ERROR 13:35:15 ERROR --- status code --- 13:35:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:35:15 ERROR --- http response --- 13:35:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:35:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:35:15 INFO [http_cmd]: http status code OK 13:35:15 DEBUG --- status code --- 13:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:15 DEBUG --- http response --- 13:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911315" 13:35:15 INFO [http_cmd]: http status code OK 13:35:15 DEBUG --- status code --- 13:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:15 DEBUG --- http response --- 13:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911315" 13:35:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911315" 13:35:15 INFO [http_cmd]: http status code OK 13:35:15 DEBUG --- status code --- 13:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:15 DEBUG --- http response --- 13:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:15 INFO [http_cmd]: http status code OK 13:35:15 DEBUG --- status code --- 13:35:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:15 DEBUG --- http response --- 13:35:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- http response --- 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735911316, "0" ] } ] } } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:35:16 DEBUG ----- output ----- 13:35:16 DEBUG { 13:35:16 DEBUG "status": "success", 13:35:16 DEBUG "data": { 13:35:16 DEBUG "resultType": "vector", 13:35:16 DEBUG "result": [ 13:35:16 DEBUG { 13:35:16 DEBUG "metric": { 13:35:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG [...] 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG "pod": "ds-cts-0" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 1735911316, 13:35:16 DEBUG "0" 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735911316, "427.64669033201073" ] } ] } } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:35:16 DEBUG ----- output ----- 13:35:16 DEBUG { 13:35:16 DEBUG "status": "success", 13:35:16 DEBUG "data": { 13:35:16 DEBUG "resultType": "vector", 13:35:16 DEBUG "result": [ 13:35:16 DEBUG { 13:35:16 DEBUG "metric": { 13:35:16 DEBUG "instance": "10.142.0.11:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG [...] 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG "instance": "10.142.0.51:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 1735911316, 13:35:16 DEBUG "0.00046666666666700016" 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:35:16 DEBUG ----- output ----- 13:35:16 DEBUG { 13:35:16 DEBUG "status": "success", 13:35:16 DEBUG "data": { 13:35:16 DEBUG "resultType": "vector", 13:35:16 DEBUG "result": [ 13:35:16 DEBUG { 13:35:16 DEBUG "metric": { 13:35:16 DEBUG "instance": "10.142.0.11:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG [...] 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG "instance": "10.142.0.51:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 1735911316, 13:35:16 DEBUG "76154.86666666667" 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:35:16 DEBUG ----- output ----- 13:35:16 DEBUG { 13:35:16 DEBUG "status": "success", 13:35:16 DEBUG "data": { 13:35:16 DEBUG "resultType": "vector", 13:35:16 DEBUG "result": [ 13:35:16 DEBUG { 13:35:16 DEBUG "metric": { 13:35:16 DEBUG "instance": "10.142.0.11:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG [...] 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG "instance": "10.142.0.51:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 1735911316, 13:35:16 DEBUG "95902.53333333331" 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:35:16 DEBUG ----- output ----- 13:35:16 DEBUG { 13:35:16 DEBUG "status": "success", 13:35:16 DEBUG "data": { 13:35:16 DEBUG "resultType": "vector", 13:35:16 DEBUG "result": [ 13:35:16 DEBUG { 13:35:16 DEBUG "metric": { 13:35:16 DEBUG "instance": "10.142.0.11:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG [...] 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG "instance": "10.142.0.51:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 1735911316, 13:35:16 DEBUG "20070.399999999998" 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:35:16 DEBUG ----- output ----- 13:35:16 DEBUG { 13:35:16 DEBUG "status": "success", 13:35:16 DEBUG "data": { 13:35:16 DEBUG "resultType": "vector", 13:35:16 DEBUG "result": [ 13:35:16 DEBUG { 13:35:16 DEBUG "metric": { 13:35:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG [...] 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 1735911316, 13:35:16 DEBUG "0.040236913479868915" 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:35:16 DEBUG ----- output ----- 13:35:16 DEBUG { 13:35:16 DEBUG "status": "success", 13:35:16 DEBUG "data": { 13:35:16 DEBUG "resultType": "vector", 13:35:16 DEBUG "result": [ 13:35:16 DEBUG { 13:35:16 DEBUG "metric": { 13:35:16 DEBUG "instance": "10.142.0.11:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG [...] 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG "instance": "10.142.0.51:9100" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 1735911316, 13:35:16 DEBUG "0" 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG } 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911316" 13:35:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911316" 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:35:16 DEBUG ----- output ----- 13:35:16 DEBUG { 13:35:16 DEBUG "status": "success", 13:35:16 DEBUG "data": { 13:35:16 DEBUG "resultType": "vector", 13:35:16 DEBUG "result": [ 13:35:16 DEBUG { 13:35:16 DEBUG "metric": { 13:35:16 DEBUG "pod": "ds-idrepo-0" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG [...] 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 1735911316, 13:35:16 DEBUG "758779904" 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG } 13:35:16 INFO [http_cmd]: http status code OK 13:35:16 DEBUG --- status code --- 13:35:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:35:16 DEBUG --- http response --- 13:35:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:35:16 DEBUG ----- output ----- 13:35:16 DEBUG { 13:35:16 DEBUG "status": "success", 13:35:16 DEBUG "data": { 13:35:16 DEBUG "resultType": "vector", 13:35:16 DEBUG "result": [ 13:35:16 DEBUG { 13:35:16 DEBUG "metric": { 13:35:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG [...] 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG 13:35:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:35:16 DEBUG }, 13:35:16 DEBUG "value": [ 13:35:16 DEBUG 1735911316, 13:35:16 DEBUG "14986608640" 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG ] 13:35:16 DEBUG } 13:35:16 DEBUG } 13:35:26 INFO 13:35:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:35:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:35:26 INFO [loop_until]: OK (rc = 0) 13:35:26 DEBUG --- stdout --- 13:35:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3850Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 93m 0% 19057Mi 67% 13:35:26 DEBUG --- stderr --- 13:35:26 DEBUG 13:35:29 INFO 13:35:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:35:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:35:29 INFO [loop_until]: OK (rc = 0) 13:35:29 DEBUG --- stdout --- 13:35:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:35:29 DEBUG --- stderr --- 13:35:29 DEBUG 13:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911375" 13:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911375" 13:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911375" 13:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911375" 13:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911375" 13:36:15 INFO [http_cmd]: http status code OK 13:36:15 DEBUG --- status code --- 13:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:15 INFO [http_cmd]: http status code OK 13:36:15 DEBUG --- http response --- 13:36:15 DEBUG --- status code --- 13:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:15 DEBUG --- http response --- 13:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:15 ERROR [http_cmd]: ERROR 13:36:15 ERROR --- status code --- 13:36:15 INFO [http_cmd]: http status code OK 13:36:15 INFO [http_cmd]: http status code OK 13:36:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:36:15 DEBUG --- status code --- 13:36:15 DEBUG --- status code --- 13:36:15 ERROR --- http response --- 13:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:36:15 DEBUG --- http response --- 13:36:15 DEBUG --- http response --- 13:36:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911375" 13:36:15 INFO [http_cmd]: http status code OK 13:36:15 DEBUG --- status code --- 13:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:15 DEBUG --- http response --- 13:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911375" 13:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911375" 13:36:15 INFO [http_cmd]: http status code OK 13:36:15 DEBUG --- status code --- 13:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:15 DEBUG --- http response --- 13:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:15 INFO [http_cmd]: http status code OK 13:36:15 DEBUG --- status code --- 13:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:15 DEBUG --- http response --- 13:36:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG --- status code --- 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG --- http response --- 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735911376, "0" ] } ] } } 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:36:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:36:16 DEBUG ----- output ----- 13:36:16 DEBUG ----- output ----- 13:36:16 DEBUG { 13:36:16 DEBUG { 13:36:16 DEBUG "status": "success", 13:36:16 DEBUG "status": "success", 13:36:16 DEBUG "data": { 13:36:16 DEBUG "data": { 13:36:16 DEBUG "resultType": "vector", 13:36:16 DEBUG "resultType": "vector", 13:36:16 DEBUG "result": [ 13:36:16 DEBUG "result": [ 13:36:16 DEBUG { 13:36:16 DEBUG { 13:36:16 DEBUG "metric": { 13:36:16 DEBUG "metric": { 13:36:16 DEBUG "instance": "10.142.0.11:9100" 13:36:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:36:16 DEBUG }, 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG [...] 13:36:16 DEBUG [...] 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-m29zn" }, "value": [ 1735911376, "352.3946085460281" ] } ] } } 13:36:16 DEBUG "instance": "10.142.0.51:9100" 13:36:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG }, 13:36:16 DEBUG }, 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG "value": [ 13:36:16 DEBUG "value": [ 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG 1735911376, 13:36:16 DEBUG 1735911376, 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG "0.00043333333333445503" 13:36:16 DEBUG "0" 13:36:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:36:16 DEBUG ] 13:36:16 DEBUG ] 13:36:16 DEBUG ----- output ----- 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:16 DEBUG { 13:36:16 DEBUG ] 13:36:16 DEBUG ] 13:36:16 DEBUG "status": "success", 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:16 DEBUG "data": { 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:16 DEBUG "resultType": "vector", 13:36:16 DEBUG "result": [ 13:36:16 DEBUG { 13:36:16 DEBUG "metric": { 13:36:16 DEBUG "instance": "10.142.0.11:9100" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG [...] 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG "instance": "10.142.0.51:9100" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 1735911376, 13:36:16 DEBUG "74994.1" 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:36:16 DEBUG ----- output ----- 13:36:16 DEBUG { 13:36:16 DEBUG "status": "success", 13:36:16 DEBUG "data": { 13:36:16 DEBUG "resultType": "vector", 13:36:16 DEBUG "result": [ 13:36:16 DEBUG { 13:36:16 DEBUG "metric": { 13:36:16 DEBUG "instance": "10.142.0.11:9100" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG [...] 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG "instance": "10.142.0.51:9100" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 1735911376, 13:36:16 DEBUG "93940.90000000001" 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:36:16 DEBUG ----- output ----- 13:36:16 DEBUG { 13:36:16 DEBUG "status": "success", 13:36:16 DEBUG "data": { 13:36:16 DEBUG "resultType": "vector", 13:36:16 DEBUG "result": [ 13:36:16 DEBUG { 13:36:16 DEBUG "metric": { 13:36:16 DEBUG "instance": "10.142.0.11:9100" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG [...] 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG "instance": "10.142.0.51:9100" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 1735911376, 13:36:16 DEBUG "25122.13333333333" 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:36:16 DEBUG ----- output ----- 13:36:16 DEBUG { 13:36:16 DEBUG "status": "success", 13:36:16 DEBUG "data": { 13:36:16 DEBUG "resultType": "vector", 13:36:16 DEBUG "result": [ 13:36:16 DEBUG { 13:36:16 DEBUG "metric": { 13:36:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG [...] 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 1735911376, 13:36:16 DEBUG "0.04622322646418531" 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:36:16 DEBUG ----- output ----- 13:36:16 DEBUG { 13:36:16 DEBUG "status": "success", 13:36:16 DEBUG "data": { 13:36:16 DEBUG "resultType": "vector", 13:36:16 DEBUG "result": [ 13:36:16 DEBUG { 13:36:16 DEBUG "metric": { 13:36:16 DEBUG "instance": "10.142.0.11:9100" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG [...] 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG "instance": "10.142.0.51:9100" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 1735911376, 13:36:16 DEBUG "0" 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911376" 13:36:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911376" 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:36:16 DEBUG ----- output ----- 13:36:16 DEBUG { 13:36:16 DEBUG "status": "success", 13:36:16 DEBUG "data": { 13:36:16 DEBUG "resultType": "vector", 13:36:16 DEBUG "result": [ 13:36:16 DEBUG { 13:36:16 DEBUG "metric": { 13:36:16 DEBUG "pod": "ds-idrepo-0" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG [...] 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 1735911376, 13:36:16 DEBUG "758779904" 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:16 INFO [http_cmd]: http status code OK 13:36:16 DEBUG --- status code --- 13:36:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:36:16 DEBUG --- http response --- 13:36:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:36:16 DEBUG ----- output ----- 13:36:16 DEBUG { 13:36:16 DEBUG "status": "success", 13:36:16 DEBUG "data": { 13:36:16 DEBUG "resultType": "vector", 13:36:16 DEBUG "result": [ 13:36:16 DEBUG { 13:36:16 DEBUG "metric": { 13:36:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG [...] 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG 13:36:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:36:16 DEBUG }, 13:36:16 DEBUG "value": [ 13:36:16 DEBUG 1735911376, 13:36:16 DEBUG "14986428416" 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG ] 13:36:16 DEBUG } 13:36:16 DEBUG } 13:36:26 INFO 13:36:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:36:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:36:26 INFO [loop_until]: OK (rc = 0) 13:36:26 DEBUG --- stdout --- 13:36:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3862Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 92m 0% 19057Mi 67% 13:36:26 DEBUG --- stderr --- 13:36:26 DEBUG 13:36:29 INFO 13:36:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:36:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:36:29 INFO [loop_until]: OK (rc = 0) 13:36:29 DEBUG --- stdout --- 13:36:29 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:36:29 DEBUG --- stderr --- 13:36:29 DEBUG 13:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911435" 13:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911435" 13:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911435" 13:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911435" 13:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911435" 13:37:15 INFO [http_cmd]: http status code OK 13:37:15 DEBUG --- status code --- 13:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:15 DEBUG --- http response --- 13:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:15 ERROR [http_cmd]: ERROR 13:37:15 INFO [http_cmd]: http status code OK 13:37:15 INFO [http_cmd]: http status code OK 13:37:15 ERROR --- status code --- 13:37:15 INFO [http_cmd]: http status code OK 13:37:15 DEBUG --- status code --- 13:37:15 DEBUG --- status code --- 13:37:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:37:15 DEBUG --- status code --- 13:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:15 ERROR --- http response --- 13:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:15 DEBUG --- http response --- 13:37:15 DEBUG --- http response --- 13:37:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:37:15 DEBUG --- http response --- 13:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911435" 13:37:15 INFO [http_cmd]: http status code OK 13:37:15 DEBUG --- status code --- 13:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:15 DEBUG --- http response --- 13:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911435" 13:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911435" 13:37:15 INFO [http_cmd]: http status code OK 13:37:15 DEBUG --- status code --- 13:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:15 DEBUG --- http response --- 13:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:15 INFO [http_cmd]: http status code OK 13:37:15 DEBUG --- status code --- 13:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:15 DEBUG --- http response --- 13:37:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-j4rsk" }, "value": [ 1735911436, "0" ] } ] } } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:37:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:37:16 DEBUG --- http response --- 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG ----- output ----- 13:37:16 DEBUG ----- output ----- 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-j4rsk" }, "value": [ 1735911436, "513.3957948171591" ] } ] } } 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG { 13:37:16 DEBUG "status": "success", 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG "data": { 13:37:16 DEBUG { 13:37:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "rcs-645c86fdd6-j4rsk" }, "value": [ 1735911436, "0.0007589070285661397" ] } ] } } 13:37:16 DEBUG "resultType": "vector", 13:37:16 DEBUG "status": "success", 13:37:16 DEBUG "result": [ 13:37:16 DEBUG "data": { 13:37:16 DEBUG { 13:37:16 DEBUG "resultType": "vector", 13:37:16 DEBUG "metric": { 13:37:16 DEBUG "result": [ 13:37:16 DEBUG "instance": "10.142.0.11:9100" 13:37:16 DEBUG { 13:37:16 DEBUG }, 13:37:16 DEBUG "metric": { 13:37:16 DEBUG "value": [ 13:37:16 DEBUG "instance": "10.142.0.11:9100" 13:37:16 DEBUG 13:37:16 DEBUG }, 13:37:16 DEBUG 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG [...] 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG [...] 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG "instance": "10.142.0.51:9100" 13:37:16 DEBUG 13:37:16 DEBUG }, 13:37:16 DEBUG 13:37:16 DEBUG "value": [ 13:37:16 DEBUG "instance": "10.142.0.51:9100" 13:37:16 DEBUG 1735911436, 13:37:16 DEBUG }, 13:37:16 DEBUG "73834.76666666666" 13:37:16 DEBUG "value": [ 13:37:16 DEBUG ] 13:37:16 DEBUG 1735911436, 13:37:16 DEBUG } 13:37:16 DEBUG "0.00043333333333066547" 13:37:16 DEBUG ] 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG } 13:37:16 DEBUG } 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:37:16 DEBUG ----- output ----- 13:37:16 DEBUG { 13:37:16 DEBUG "status": "success", 13:37:16 DEBUG "data": { 13:37:16 DEBUG "resultType": "vector", 13:37:16 DEBUG "result": [ 13:37:16 DEBUG { 13:37:16 DEBUG "metric": { 13:37:16 DEBUG "instance": "10.142.0.11:9100" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG [...] 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG "instance": "10.142.0.51:9100" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 1735911436, 13:37:16 DEBUG "92905.33333333333" 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:37:16 DEBUG ----- output ----- 13:37:16 DEBUG { 13:37:16 DEBUG "status": "success", 13:37:16 DEBUG "data": { 13:37:16 DEBUG "resultType": "vector", 13:37:16 DEBUG "result": [ 13:37:16 DEBUG { 13:37:16 DEBUG "metric": { 13:37:16 DEBUG "instance": "10.142.0.11:9100" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG [...] 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG "instance": "10.142.0.51:9100" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 1735911436, 13:37:16 DEBUG "20206.93333333333" 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:37:16 DEBUG ----- output ----- 13:37:16 DEBUG { 13:37:16 DEBUG "status": "success", 13:37:16 DEBUG "data": { 13:37:16 DEBUG "resultType": "vector", 13:37:16 DEBUG "result": [ 13:37:16 DEBUG { 13:37:16 DEBUG "metric": { 13:37:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG [...] 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 1735911436, 13:37:16 DEBUG "0.0403193569479598" 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:37:16 DEBUG ----- output ----- 13:37:16 DEBUG { 13:37:16 DEBUG "status": "success", 13:37:16 DEBUG "data": { 13:37:16 DEBUG "resultType": "vector", 13:37:16 DEBUG "result": [ 13:37:16 DEBUG { 13:37:16 DEBUG "metric": { 13:37:16 DEBUG "instance": "10.142.0.11:9100" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG [...] 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG "instance": "10.142.0.51:9100" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 1735911436, 13:37:16 DEBUG "0.0006666666666660604" 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG } 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911436" 13:37:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911436" 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:37:16 DEBUG ----- output ----- 13:37:16 DEBUG { 13:37:16 DEBUG "status": "success", 13:37:16 DEBUG "data": { 13:37:16 DEBUG "resultType": "vector", 13:37:16 DEBUG "result": [ 13:37:16 DEBUG { 13:37:16 DEBUG "metric": { 13:37:16 DEBUG "pod": "ds-idrepo-0" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG [...] 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 1735911436, 13:37:16 DEBUG "758779904" 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG } 13:37:16 INFO [http_cmd]: http status code OK 13:37:16 DEBUG --- status code --- 13:37:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:37:16 DEBUG --- http response --- 13:37:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:37:16 DEBUG ----- output ----- 13:37:16 DEBUG { 13:37:16 DEBUG "status": "success", 13:37:16 DEBUG "data": { 13:37:16 DEBUG "resultType": "vector", 13:37:16 DEBUG "result": [ 13:37:16 DEBUG { 13:37:16 DEBUG "metric": { 13:37:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG [...] 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG 13:37:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:37:16 DEBUG }, 13:37:16 DEBUG "value": [ 13:37:16 DEBUG 1735911436, 13:37:16 DEBUG "14991794176" 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG ] 13:37:16 DEBUG } 13:37:16 DEBUG } 13:37:26 INFO 13:37:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:37:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:37:26 INFO [loop_until]: OK (rc = 0) 13:37:26 DEBUG --- stdout --- 13:37:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 109m 0% 3854Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 94m 0% 19058Mi 67% 13:37:26 DEBUG --- stderr --- 13:37:26 DEBUG 13:37:29 INFO 13:37:29 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:37:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:37:30 INFO [loop_until]: OK (rc = 0) 13:37:30 DEBUG --- stdout --- 13:37:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:37:30 DEBUG --- stderr --- 13:37:30 DEBUG 13:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911495" 13:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911495" 13:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911495" 13:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911495" 13:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911495" 13:38:15 INFO [http_cmd]: http status code OK 13:38:15 DEBUG --- status code --- 13:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:15 DEBUG --- http response --- 13:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:15 INFO [http_cmd]: http status code OK 13:38:15 DEBUG --- status code --- 13:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:15 DEBUG --- http response --- 13:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:15 INFO [http_cmd]: http status code OK 13:38:15 DEBUG --- status code --- 13:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:15 DEBUG --- http response --- 13:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:15 ERROR [http_cmd]: ERROR 13:38:15 ERROR --- status code --- 13:38:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:38:15 ERROR --- http response --- 13:38:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:38:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:38:15 INFO [http_cmd]: http status code OK 13:38:15 DEBUG --- status code --- 13:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:15 DEBUG --- http response --- 13:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911495" 13:38:15 INFO [http_cmd]: http status code OK 13:38:15 DEBUG --- status code --- 13:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:15 DEBUG --- http response --- 13:38:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911495" 13:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911495" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG { 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "data": { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG { 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "pod": "ds-cts-0" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG "0" 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911496" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG { 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "data": { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG { 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "pod": "ds-cts-0" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG "0.0006864854031649537" 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG { 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "data": { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG { 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "instance": "10.142.0.11:9100" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "instance": "10.142.0.51:9100" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG "0.00036666666666557526" 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG { 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG "data": { 13:38:16 DEBUG { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG "data": { 13:38:16 DEBUG { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "result": [ 13:38:16 DEBUG "pod": "ds-cts-0" 13:38:16 DEBUG { 13:38:16 DEBUG }, 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "value": [ 13:38:16 DEBUG "instance": "10.142.0.11:9100" 13:38:16 DEBUG 13:38:16 DEBUG }, 13:38:16 DEBUG 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:38:16 DEBUG 13:38:16 DEBUG }, 13:38:16 DEBUG 13:38:16 DEBUG "value": [ 13:38:16 DEBUG "instance": "10.142.0.51:9100" 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG }, 13:38:16 DEBUG "136.81608657892977" 13:38:16 DEBUG "value": [ 13:38:16 DEBUG ] 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG } 13:38:16 DEBUG "74645.70000000001" 13:38:16 DEBUG ] 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911496" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG { 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "data": { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG { 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "instance": "10.142.0.11:9100" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "instance": "10.142.0.51:9100" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG "94269.56666666665" 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911496" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG { 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "data": { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG { 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "instance": "10.142.0.11:9100" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "instance": "10.142.0.51:9100" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG "6690.133333333333" 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG { 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "data": { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG { 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG "0.04287170531929858" 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911496" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG { 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "data": { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG { 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "instance": "10.142.0.11:9100" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "instance": "10.142.0.51:9100" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG "0" 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911496" 13:38:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911496" 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG { 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "data": { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG { 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "pod": "ds-idrepo-0" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG "758779904" 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:16 INFO [http_cmd]: http status code OK 13:38:16 DEBUG --- status code --- 13:38:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:38:16 DEBUG --- http response --- 13:38:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:38:16 DEBUG ----- output ----- 13:38:16 DEBUG { 13:38:16 DEBUG "status": "success", 13:38:16 DEBUG "data": { 13:38:16 DEBUG "resultType": "vector", 13:38:16 DEBUG "result": [ 13:38:16 DEBUG { 13:38:16 DEBUG "metric": { 13:38:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG [...] 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG 13:38:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:38:16 DEBUG }, 13:38:16 DEBUG "value": [ 13:38:16 DEBUG 1735911496, 13:38:16 DEBUG "14991802368" 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG ] 13:38:16 DEBUG } 13:38:16 DEBUG } 13:38:26 INFO 13:38:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:38:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:38:26 INFO [loop_until]: OK (rc = 0) 13:38:26 DEBUG --- stdout --- 13:38:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3816Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 87m 0% 19065Mi 67% 13:38:26 DEBUG --- stderr --- 13:38:26 DEBUG 13:38:30 INFO 13:38:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:38:30 INFO [loop_until]: OK (rc = 0) 13:38:30 DEBUG --- stdout --- 13:38:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:38:30 DEBUG --- stderr --- 13:38:30 DEBUG 13:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911555" 13:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911555" 13:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911555" 13:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911555" 13:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911555" 13:39:15 INFO [http_cmd]: http status code OK 13:39:15 INFO [http_cmd]: http status code OK 13:39:15 INFO [http_cmd]: http status code OK 13:39:15 DEBUG --- status code --- 13:39:15 DEBUG --- status code --- 13:39:15 DEBUG --- status code --- 13:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:15 DEBUG --- http response --- 13:39:15 DEBUG --- http response --- 13:39:15 DEBUG --- http response --- 13:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:15 ERROR [http_cmd]: ERROR 13:39:15 INFO [http_cmd]: http status code OK 13:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:15 ERROR --- status code --- 13:39:15 DEBUG --- status code --- 13:39:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:15 ERROR --- http response --- 13:39:15 DEBUG --- http response --- 13:39:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911555" 13:39:15 INFO [http_cmd]: http status code OK 13:39:15 DEBUG --- status code --- 13:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:15 DEBUG --- http response --- 13:39:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG --- status code --- 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "pod": "ds-cts-0" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "0" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "pod": "ds-cts-0" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "0.002674301058288157" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "instance": "10.142.0.11:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "instance": "10.142.0.51:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "0.0003999999999981204" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "instance": "10.142.0.11:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "instance": "10.142.0.51:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "76088.70000000001" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "pod": "ds-cts-0" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "517.5801611119886" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "instance": "10.142.0.11:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "instance": "10.142.0.51:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "95702.70000000003" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "instance": "10.142.0.11:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "instance": "10.142.0.51:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "23483.733333333334" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "0.044167950167742775" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "instance": "10.142.0.11:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "instance": "10.142.0.51:9100" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "0.00033333333333350387" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911556" 13:39:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911556" 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "pod": "ds-idrepo-0" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "758779904" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:16 INFO [http_cmd]: http status code OK 13:39:16 DEBUG --- status code --- 13:39:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:39:16 DEBUG --- http response --- 13:39:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:39:16 DEBUG ----- output ----- 13:39:16 DEBUG { 13:39:16 DEBUG "status": "success", 13:39:16 DEBUG "data": { 13:39:16 DEBUG "resultType": "vector", 13:39:16 DEBUG "result": [ 13:39:16 DEBUG { 13:39:16 DEBUG "metric": { 13:39:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG [...] 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG 13:39:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:39:16 DEBUG }, 13:39:16 DEBUG "value": [ 13:39:16 DEBUG 1735911556, 13:39:16 DEBUG "14990897152" 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG ] 13:39:16 DEBUG } 13:39:16 DEBUG } 13:39:26 INFO 13:39:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:39:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:39:26 INFO [loop_until]: OK (rc = 0) 13:39:26 DEBUG --- stdout --- 13:39:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 115m 0% 3847Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 94m 0% 19061Mi 67% 13:39:26 DEBUG --- stderr --- 13:39:26 DEBUG 13:39:30 INFO 13:39:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:39:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:39:30 INFO [loop_until]: OK (rc = 0) 13:39:30 DEBUG --- stdout --- 13:39:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:39:30 DEBUG --- stderr --- 13:39:30 DEBUG 13:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911615" 13:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911615" 13:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911615" 13:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911615" 13:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911615" 13:40:15 INFO [http_cmd]: http status code OK 13:40:15 DEBUG --- status code --- 13:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:15 DEBUG --- http response --- 13:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:15 INFO [http_cmd]: http status code OK 13:40:15 DEBUG --- status code --- 13:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:15 DEBUG --- http response --- 13:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:15 INFO [http_cmd]: http status code OK 13:40:15 DEBUG --- status code --- 13:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:15 DEBUG --- http response --- 13:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:15 ERROR [http_cmd]: ERROR 13:40:15 INFO [http_cmd]: http status code OK 13:40:15 ERROR --- status code --- 13:40:15 DEBUG --- status code --- 13:40:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:15 ERROR --- http response --- 13:40:15 DEBUG --- http response --- 13:40:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911615" 13:40:15 INFO [http_cmd]: http status code OK 13:40:15 DEBUG --- status code --- 13:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:15 DEBUG --- http response --- 13:40:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG { 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG "data": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "result": [ 13:40:16 DEBUG { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "pod": "ds-cts-0" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG "0.000632214811953424" 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG ] 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG } 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:40:16 DEBUG ] 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG } 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG { 13:40:16 DEBUG } 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:40:16 DEBUG "data": { 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG { 13:40:16 DEBUG "result": [ 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG { 13:40:16 DEBUG "data": { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "pod": "ds-cts-0" 13:40:16 DEBUG "result": [ 13:40:16 DEBUG }, 13:40:16 DEBUG { 13:40:16 DEBUG "value": [ 13:40:16 DEBUG "metric": { 13:40:16 DEBUG 13:40:16 DEBUG "instance": "10.142.0.11:9100" 13:40:16 DEBUG 13:40:16 DEBUG }, 13:40:16 DEBUG 13:40:16 DEBUG "value": [ 13:40:16 DEBUG [...] 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911616" 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:40:16 DEBUG 13:40:16 DEBUG }, 13:40:16 DEBUG 13:40:16 DEBUG "value": [ 13:40:16 DEBUG "instance": "10.142.0.51:9100" 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG }, 13:40:16 DEBUG "0" 13:40:16 DEBUG "value": [ 13:40:16 DEBUG ] 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911616" 13:40:16 DEBUG "0.00046666666667552665" 13:40:16 DEBUG ] 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG { 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG "data": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "result": [ 13:40:16 DEBUG { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "pod": "ds-cts-0" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG "286.0435071056951" 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG { 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG "data": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "result": [ 13:40:16 DEBUG { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "instance": "10.142.0.11:9100" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG "instance": "10.142.0.51:9100" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG "76992.53333333333" 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG { 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG "data": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "result": [ 13:40:16 DEBUG { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "instance": "10.142.0.11:9100" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG "instance": "10.142.0.51:9100" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG "96508.43333333333" 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG { 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG "data": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "result": [ 13:40:16 DEBUG { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "instance": "10.142.0.11:9100" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG "instance": "10.142.0.51:9100" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG "22937.6" 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG { 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG "data": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "result": [ 13:40:16 DEBUG { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG "0.045206909514965504" 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG { 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG "data": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "result": [ 13:40:16 DEBUG { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "instance": "10.142.0.11:9100" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG "instance": "10.142.0.51:9100" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG "0.0006666666666660604" 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911616" 13:40:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911616" 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG { 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG "data": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "result": [ 13:40:16 DEBUG { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "pod": "ds-idrepo-0" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG "758779904" 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:16 INFO [http_cmd]: http status code OK 13:40:16 DEBUG --- status code --- 13:40:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:40:16 DEBUG --- http response --- 13:40:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:40:16 DEBUG ----- output ----- 13:40:16 DEBUG { 13:40:16 DEBUG "status": "success", 13:40:16 DEBUG "data": { 13:40:16 DEBUG "resultType": "vector", 13:40:16 DEBUG "result": [ 13:40:16 DEBUG { 13:40:16 DEBUG "metric": { 13:40:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG [...] 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG 13:40:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:40:16 DEBUG }, 13:40:16 DEBUG "value": [ 13:40:16 DEBUG 1735911616, 13:40:16 DEBUG "14991396864" 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG ] 13:40:16 DEBUG } 13:40:16 DEBUG } 13:40:26 INFO 13:40:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:40:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:40:26 INFO [loop_until]: OK (rc = 0) 13:40:26 DEBUG --- stdout --- 13:40:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 117m 0% 3807Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 90m 0% 19060Mi 67% 13:40:26 DEBUG --- stderr --- 13:40:26 DEBUG 13:40:30 INFO 13:40:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:40:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:40:30 INFO [loop_until]: OK (rc = 0) 13:40:30 DEBUG --- stdout --- 13:40:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:40:30 DEBUG --- stderr --- 13:40:30 DEBUG 13:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911675" 13:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911675" 13:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911675" 13:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911675" 13:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911675" 13:41:15 INFO [http_cmd]: http status code OK 13:41:15 DEBUG --- status code --- 13:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:15 DEBUG --- http response --- 13:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:15 INFO [http_cmd]: http status code OK 13:41:15 DEBUG --- status code --- 13:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:15 DEBUG --- http response --- 13:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:15 INFO [http_cmd]: http status code OK 13:41:15 DEBUG --- status code --- 13:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:15 DEBUG --- http response --- 13:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:15 INFO [http_cmd]: http status code OK 13:41:15 DEBUG --- status code --- 13:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:15 DEBUG --- http response --- 13:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:15 ERROR [http_cmd]: ERROR 13:41:15 ERROR --- status code --- 13:41:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:41:15 ERROR --- http response --- 13:41:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:41:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911675" 13:41:15 INFO [http_cmd]: http status code OK 13:41:15 DEBUG --- status code --- 13:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:15 DEBUG --- http response --- 13:41:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:41:16 DEBUG ----- output ----- 13:41:16 DEBUG { 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG "status": "success", 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG "data": { 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG "resultType": "vector", 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG "result": [ 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:41:16 DEBUG { 13:41:16 DEBUG "metric": { 13:41:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG [...] 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG "pod": "ds-cts-0" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 1735911676, 13:41:16 DEBUG "0.0027436529455287296" 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735911676, "0" ] } ] } } 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:41:16 DEBUG ----- output ----- 13:41:16 DEBUG { 13:41:16 DEBUG "status": "success", 13:41:16 DEBUG "data": { 13:41:16 DEBUG "resultType": "vector", 13:41:16 DEBUG "result": [ 13:41:16 DEBUG { 13:41:16 DEBUG "metric": { 13:41:16 DEBUG "instance": "10.142.0.11:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG [...] 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG "instance": "10.142.0.51:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 1735911676, 13:41:16 DEBUG "0.0005333333333330378" 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG } 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:41:16 DEBUG ----- output ----- 13:41:16 DEBUG { 13:41:16 DEBUG "status": "success", 13:41:16 DEBUG "data": { 13:41:16 DEBUG "resultType": "vector", 13:41:16 DEBUG "result": [ 13:41:16 DEBUG { 13:41:16 DEBUG "metric": { 13:41:16 DEBUG "instance": "10.142.0.11:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [...] 13:41:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-cts-0" }, "value": [ 1735911676, "432.69129194689947" ] } ] } } 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG "instance": "10.142.0.51:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 1735911676, 13:41:16 DEBUG "74580.59999999999" 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:41:16 DEBUG ----- output ----- 13:41:16 DEBUG { 13:41:16 DEBUG "status": "success", 13:41:16 DEBUG "data": { 13:41:16 DEBUG "resultType": "vector", 13:41:16 DEBUG "result": [ 13:41:16 DEBUG { 13:41:16 DEBUG "metric": { 13:41:16 DEBUG "instance": "10.142.0.11:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG [...] 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG "instance": "10.142.0.51:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 1735911676, 13:41:16 DEBUG "93671.5" 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:41:16 DEBUG ----- output ----- 13:41:16 DEBUG { 13:41:16 DEBUG "status": "success", 13:41:16 DEBUG "data": { 13:41:16 DEBUG "resultType": "vector", 13:41:16 DEBUG "result": [ 13:41:16 DEBUG { 13:41:16 DEBUG "metric": { 13:41:16 DEBUG "instance": "10.142.0.11:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG [...] 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG "instance": "10.142.0.51:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 1735911676, 13:41:16 DEBUG "20480" 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG } 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:41:16 DEBUG ----- output ----- 13:41:16 DEBUG { 13:41:16 DEBUG "status": "success", 13:41:16 DEBUG "data": { 13:41:16 DEBUG "resultType": "vector", 13:41:16 DEBUG "result": [ 13:41:16 DEBUG { 13:41:16 DEBUG "metric": { 13:41:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG [...] 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 1735911676, 13:41:16 DEBUG "0.03889860337298061" 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:41:16 DEBUG ----- output ----- 13:41:16 DEBUG { 13:41:16 DEBUG "status": "success", 13:41:16 DEBUG "data": { 13:41:16 DEBUG "resultType": "vector", 13:41:16 DEBUG "result": [ 13:41:16 DEBUG { 13:41:16 DEBUG "metric": { 13:41:16 DEBUG "instance": "10.142.0.11:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG [...] 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG "instance": "10.142.0.51:9100" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 1735911676, 13:41:16 DEBUG "0.0003333333333330302" 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG } 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911676" 13:41:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911676" 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:41:16 DEBUG ----- output ----- 13:41:16 DEBUG { 13:41:16 DEBUG "status": "success", 13:41:16 DEBUG "data": { 13:41:16 DEBUG "resultType": "vector", 13:41:16 DEBUG "result": [ 13:41:16 DEBUG { 13:41:16 DEBUG "metric": { 13:41:16 DEBUG "pod": "ds-idrepo-0" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG [...] 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 1735911676, 13:41:16 DEBUG "758779904" 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG } 13:41:16 INFO [http_cmd]: http status code OK 13:41:16 DEBUG --- status code --- 13:41:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:41:16 DEBUG --- http response --- 13:41:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:41:16 DEBUG ----- output ----- 13:41:16 DEBUG { 13:41:16 DEBUG "status": "success", 13:41:16 DEBUG "data": { 13:41:16 DEBUG "resultType": "vector", 13:41:16 DEBUG "result": [ 13:41:16 DEBUG { 13:41:16 DEBUG "metric": { 13:41:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG [...] 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG 13:41:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:41:16 DEBUG }, 13:41:16 DEBUG "value": [ 13:41:16 DEBUG 1735911676, 13:41:16 DEBUG "14986780672" 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG ] 13:41:16 DEBUG } 13:41:16 DEBUG } 13:41:26 INFO 13:41:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:41:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:41:26 INFO [loop_until]: OK (rc = 0) 13:41:26 DEBUG --- stdout --- 13:41:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 113m 0% 3821Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 91m 0% 19051Mi 66% 13:41:26 DEBUG --- stderr --- 13:41:26 DEBUG 13:41:30 INFO 13:41:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:41:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:41:30 INFO [loop_until]: OK (rc = 0) 13:41:30 DEBUG --- stdout --- 13:41:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:41:30 DEBUG --- stderr --- 13:41:30 DEBUG 13:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911735" 13:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911735" 13:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911735" 13:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911735" 13:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911735" 13:42:15 INFO [http_cmd]: http status code OK 13:42:15 DEBUG --- status code --- 13:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:15 DEBUG --- http response --- 13:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:15 ERROR [http_cmd]: ERROR 13:42:15 INFO [http_cmd]: http status code OK 13:42:15 ERROR --- status code --- 13:42:15 DEBUG --- status code --- 13:42:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:15 ERROR --- http response --- 13:42:15 DEBUG --- http response --- 13:42:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:42:15 INFO [http_cmd]: http status code OK 13:42:15 INFO [http_cmd]: http status code OK 13:42:15 DEBUG --- status code --- 13:42:15 DEBUG --- status code --- 13:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:15 DEBUG --- http response --- 13:42:15 DEBUG --- http response --- 13:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911735" 13:42:15 INFO [http_cmd]: http status code OK 13:42:15 DEBUG --- status code --- 13:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:15 DEBUG --- http response --- 13:42:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 DEBUG { 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG [...] 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG "pod": "ds-cts-0" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "0.0027164922090710856" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911736" 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911736" 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 DEBUG { 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "instance": "10.142.0.11:9100" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG [...] 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG "instance": "10.142.0.51:9100" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "0.0013333333333368576" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 DEBUG { 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG [...] 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG "pod": "ds-cts-0" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "0" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG { 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG "metric": { 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG "instance": "10.142.0.11:9100" 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG }, 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:42:16 DEBUG "value": [ 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG 13:42:16 DEBUG { 13:42:16 DEBUG 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG 13:42:16 DEBUG "data": { 13:42:16 DEBUG [...] 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG 13:42:16 DEBUG "result": [ 13:42:16 DEBUG 13:42:16 DEBUG { 13:42:16 DEBUG 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "instance": "10.142.0.51:9100" 13:42:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:42:16 DEBUG }, 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG 13:42:16 DEBUG "74492.66666666667" 13:42:16 DEBUG 13:42:16 DEBUG ] 13:42:16 DEBUG 13:42:16 DEBUG } 13:42:16 DEBUG [...] 13:42:16 DEBUG ] 13:42:16 DEBUG 13:42:16 DEBUG } 13:42:16 DEBUG 13:42:16 DEBUG } 13:42:16 DEBUG 13:42:16 DEBUG "pod": "ds-cts-0" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "558.6279791332811" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 DEBUG { 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "instance": "10.142.0.11:9100" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG [...] 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG "instance": "10.142.0.51:9100" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "93637.66666666666" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 DEBUG { 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "instance": "10.142.0.11:9100" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG [...] 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG "instance": "10.142.0.51:9100" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "24439.466666666667" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 DEBUG { 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG [...] 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "0.03953056257078181" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 DEBUG { 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "instance": "10.142.0.11:9100" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG [...] 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG "instance": "10.142.0.51:9100" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "0.001000000000000038" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911736" 13:42:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911736" 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 DEBUG { 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "pod": "ds-idrepo-0" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG [...] 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "758779904" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:16 INFO [http_cmd]: http status code OK 13:42:16 DEBUG --- status code --- 13:42:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:42:16 DEBUG --- http response --- 13:42:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:42:16 DEBUG ----- output ----- 13:42:16 DEBUG { 13:42:16 DEBUG "status": "success", 13:42:16 DEBUG "data": { 13:42:16 DEBUG "resultType": "vector", 13:42:16 DEBUG "result": [ 13:42:16 DEBUG { 13:42:16 DEBUG "metric": { 13:42:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG [...] 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG 13:42:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:42:16 DEBUG }, 13:42:16 DEBUG "value": [ 13:42:16 DEBUG 1735911736, 13:42:16 DEBUG "14987165696" 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG ] 13:42:16 DEBUG } 13:42:16 DEBUG } 13:42:26 INFO 13:42:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:42:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:42:26 INFO [loop_until]: OK (rc = 0) 13:42:26 DEBUG --- stdout --- 13:42:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 112m 0% 3844Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 90m 0% 19057Mi 67% 13:42:26 DEBUG --- stderr --- 13:42:26 DEBUG 13:42:30 INFO 13:42:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:42:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:42:30 INFO [loop_until]: OK (rc = 0) 13:42:30 DEBUG --- stdout --- 13:42:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:42:30 DEBUG --- stderr --- 13:42:30 DEBUG 13:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911795" 13:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911795" 13:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911795" 13:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911795" 13:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911795" 13:43:15 INFO [http_cmd]: http status code OK 13:43:15 INFO [http_cmd]: http status code OK 13:43:15 INFO [http_cmd]: http status code OK 13:43:15 ERROR [http_cmd]: ERROR 13:43:15 DEBUG --- status code --- 13:43:15 DEBUG --- status code --- 13:43:15 DEBUG --- status code --- 13:43:15 ERROR --- status code --- 13:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:43:15 DEBUG --- http response --- 13:43:15 DEBUG --- http response --- 13:43:15 DEBUG --- http response --- 13:43:15 ERROR --- http response --- 13:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:43:15 INFO [http_cmd]: http status code OK 13:43:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:43:15 DEBUG --- status code --- 13:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:15 DEBUG --- http response --- 13:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911795" 13:43:15 INFO [http_cmd]: http status code OK 13:43:15 DEBUG --- status code --- 13:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:15 DEBUG --- http response --- 13:43:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG --- status code --- 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG { 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "data": { 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "result": [ 13:43:16 DEBUG { 13:43:16 DEBUG "metric": { 13:43:16 DEBUG "pod": "ds-cts-0" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG "pod": "ds-idrepo-0" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG "0.0023146800869081104" 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG { 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "data": { 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "result": [ 13:43:16 DEBUG { 13:43:16 DEBUG "metric": { 13:43:16 DEBUG "pod": "ds-cts-0" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG "pod": "ds-idrepo-0" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG "0" 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:43:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG { 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "data": { 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "result": [ 13:43:16 DEBUG { 13:43:16 DEBUG "metric": { 13:43:16 DEBUG "instance": "10.142.0.11:9100" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG "instance": "10.142.0.51:9100" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG "0.0004000000000019099" 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:43:16 DEBUG { 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG { 13:43:16 DEBUG "data": { 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "data": { 13:43:16 DEBUG "result": [ 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG { 13:43:16 DEBUG "result": [ 13:43:16 DEBUG "metric": { 13:43:16 DEBUG { 13:43:16 DEBUG "pod": "ds-cts-0" 13:43:16 DEBUG "metric": { 13:43:16 DEBUG }, 13:43:16 DEBUG "instance": "10.142.0.11:9100" 13:43:16 DEBUG "value": [ 13:43:16 DEBUG }, 13:43:16 DEBUG 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG "pod": "ds-idrepo-0" 13:43:16 DEBUG 13:43:16 DEBUG }, 13:43:16 DEBUG "instance": "10.142.0.51:9100" 13:43:16 DEBUG "value": [ 13:43:16 DEBUG }, 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG "318.68671048871107" 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG ] 13:43:16 DEBUG "73631" 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG { 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "data": { 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "result": [ 13:43:16 DEBUG { 13:43:16 DEBUG "metric": { 13:43:16 DEBUG "instance": "10.142.0.11:9100" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG "instance": "10.142.0.51:9100" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG "93417.03333333334" 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911796" 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG { 13:43:16 DEBUG { 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "data": { 13:43:16 DEBUG "data": { 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "result": [ 13:43:16 DEBUG "result": [ 13:43:16 DEBUG { 13:43:16 DEBUG { 13:43:16 DEBUG "metric": { 13:43:16 DEBUG "metric": { 13:43:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:43:16 DEBUG "instance": "10.142.0.11:9100" 13:43:16 DEBUG }, 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:43:16 DEBUG "instance": "10.142.0.51:9100" 13:43:16 DEBUG }, 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG "0.039032786352486075" 13:43:16 DEBUG "23483.733333333334" 13:43:16 DEBUG ] 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG { 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "data": { 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "result": [ 13:43:16 DEBUG { 13:43:16 DEBUG "metric": { 13:43:16 DEBUG "instance": "10.142.0.11:9100" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG "instance": "10.142.0.51:9100" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG "0" 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911796" 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911796" 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG { 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "data": { 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "result": [ 13:43:16 DEBUG { 13:43:16 DEBUG "metric": { 13:43:16 DEBUG "pod": "ds-idrepo-0" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG "758779904" 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:16 INFO [http_cmd]: http status code OK 13:43:16 DEBUG --- status code --- 13:43:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:43:16 DEBUG --- http response --- 13:43:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:43:16 DEBUG ----- output ----- 13:43:16 DEBUG { 13:43:16 DEBUG "status": "success", 13:43:16 DEBUG "data": { 13:43:16 DEBUG "resultType": "vector", 13:43:16 DEBUG "result": [ 13:43:16 DEBUG { 13:43:16 DEBUG "metric": { 13:43:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG [...] 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG 13:43:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:43:16 DEBUG }, 13:43:16 DEBUG "value": [ 13:43:16 DEBUG 1735911796, 13:43:16 DEBUG "14989082624" 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG ] 13:43:16 DEBUG } 13:43:16 DEBUG } 13:43:26 INFO 13:43:26 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:43:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:43:27 INFO [loop_until]: OK (rc = 0) 13:43:27 DEBUG --- stdout --- 13:43:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 110m 0% 3829Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 96m 0% 19058Mi 67% 13:43:27 DEBUG --- stderr --- 13:43:27 DEBUG 13:43:30 INFO 13:43:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:43:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:43:30 INFO [loop_until]: OK (rc = 0) 13:43:30 DEBUG --- stdout --- 13:43:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:43:30 DEBUG --- stderr --- 13:43:30 DEBUG 13:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911855" 13:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911855" 13:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911855" 13:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911855" 13:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911855" 13:44:15 INFO [http_cmd]: http status code OK 13:44:15 DEBUG --- status code --- 13:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:15 DEBUG --- http response --- 13:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:15 INFO [http_cmd]: http status code OK 13:44:15 INFO [http_cmd]: http status code OK 13:44:15 ERROR [http_cmd]: ERROR 13:44:15 INFO [http_cmd]: http status code OK 13:44:15 DEBUG --- status code --- 13:44:15 DEBUG --- status code --- 13:44:15 ERROR --- status code --- 13:44:15 DEBUG --- status code --- 13:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:15 DEBUG --- http response --- 13:44:15 DEBUG --- http response --- 13:44:15 ERROR --- http response --- 13:44:15 DEBUG --- http response --- 13:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911855" 13:44:15 INFO [http_cmd]: http status code OK 13:44:15 DEBUG --- status code --- 13:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:15 DEBUG --- http response --- 13:44:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG { 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG "data": { 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG "result": [ 13:44:16 DEBUG { 13:44:16 DEBUG "metric": { 13:44:16 DEBUG "pod": "ds-idrepo-0" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG [...] 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911856" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG "0.000761638205847307" 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG { 13:44:16 DEBUG { 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG "data": { 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG "data": { 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG "result": [ 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG { 13:44:16 DEBUG "metric": { 13:44:16 DEBUG "instance": "10.142.0.11:9100" 13:44:16 DEBUG "result": [ 13:44:16 DEBUG }, 13:44:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:44:16 DEBUG { 13:44:16 DEBUG "value": [ 13:44:16 DEBUG "metric": { 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "pod": "ds-idrepo-0" 13:44:16 DEBUG 13:44:16 DEBUG }, 13:44:16 DEBUG [...] 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "instance": "10.142.0.51:9100" 13:44:16 DEBUG [...] 13:44:16 DEBUG }, 13:44:16 DEBUG 13:44:16 DEBUG "value": [ 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911856" 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 1735911856, 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911856" 13:44:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:44:16 DEBUG "0.00030000000000048505" 13:44:16 DEBUG }, 13:44:16 DEBUG ] 13:44:16 DEBUG "value": [ 13:44:16 DEBUG } 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG ] 13:44:16 DEBUG "0" 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG { 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG "data": { 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG "result": [ 13:44:16 DEBUG { 13:44:16 DEBUG "metric": { 13:44:16 DEBUG "pod": "ds-idrepo-0" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG [...] 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG "435.34330050308233" 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG { 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG "data": { 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG "result": [ 13:44:16 DEBUG { 13:44:16 DEBUG "metric": { 13:44:16 DEBUG "instance": "10.142.0.11:9100" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG [...] 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "instance": "10.142.0.51:9100" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG "76374.46666666666" 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG { 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG "data": { 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG "result": [ 13:44:16 DEBUG { 13:44:16 DEBUG "metric": { 13:44:16 DEBUG "instance": "10.142.0.11:9100" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG [...] 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "instance": "10.142.0.51:9100" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG "96082.56666666667" 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:44:16 DEBUG { 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG { 13:44:16 DEBUG "data": { 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG "data": { 13:44:16 DEBUG "result": [ 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG { 13:44:16 DEBUG "result": [ 13:44:16 DEBUG "metric": { 13:44:16 DEBUG { 13:44:16 DEBUG "instance": "10.142.0.11:9100" 13:44:16 DEBUG "metric": { 13:44:16 DEBUG }, 13:44:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:44:16 DEBUG "value": [ 13:44:16 DEBUG }, 13:44:16 DEBUG 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG [...] 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG [...] 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "instance": "10.142.0.51:9100" 13:44:16 DEBUG 13:44:16 DEBUG }, 13:44:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:44:16 DEBUG "value": [ 13:44:16 DEBUG }, 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG "6826.666666666667" 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG ] 13:44:16 DEBUG "0.04316449533435888" 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG { 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG "data": { 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG "result": [ 13:44:16 DEBUG { 13:44:16 DEBUG "metric": { 13:44:16 DEBUG "instance": "10.142.0.11:9100" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG [...] 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "instance": "10.142.0.51:9100" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG "0.0003333333333330302" 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911856" 13:44:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911856" 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG { 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG "data": { 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG "result": [ 13:44:16 DEBUG { 13:44:16 DEBUG "metric": { 13:44:16 DEBUG "pod": "ds-idrepo-0" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG [...] 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG "758779904" 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:16 INFO [http_cmd]: http status code OK 13:44:16 DEBUG --- status code --- 13:44:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:44:16 DEBUG --- http response --- 13:44:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:44:16 DEBUG ----- output ----- 13:44:16 DEBUG { 13:44:16 DEBUG "status": "success", 13:44:16 DEBUG "data": { 13:44:16 DEBUG "resultType": "vector", 13:44:16 DEBUG "result": [ 13:44:16 DEBUG { 13:44:16 DEBUG "metric": { 13:44:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG [...] 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG 13:44:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:44:16 DEBUG }, 13:44:16 DEBUG "value": [ 13:44:16 DEBUG 1735911856, 13:44:16 DEBUG "14987603968" 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG ] 13:44:16 DEBUG } 13:44:16 DEBUG } 13:44:27 INFO 13:44:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:44:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:44:27 INFO [loop_until]: OK (rc = 0) 13:44:27 DEBUG --- stdout --- 13:44:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 109m 0% 3841Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 73m 0% 19056Mi 67% 13:44:27 DEBUG --- stderr --- 13:44:27 DEBUG 13:44:30 INFO 13:44:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:44:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:44:30 INFO [loop_until]: OK (rc = 0) 13:44:30 DEBUG --- stdout --- 13:44:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:44:30 DEBUG --- stderr --- 13:44:30 DEBUG 13:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911915" 13:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911915" 13:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911915" 13:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911915" 13:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911915" 13:45:15 INFO [http_cmd]: http status code OK 13:45:15 ERROR [http_cmd]: ERROR 13:45:15 DEBUG --- status code --- 13:45:15 ERROR --- status code --- 13:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:45:15 DEBUG --- http response --- 13:45:15 ERROR --- http response --- 13:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:45:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:45:15 INFO [http_cmd]: http status code OK 13:45:15 INFO [http_cmd]: http status code OK 13:45:15 DEBUG --- status code --- 13:45:15 INFO [http_cmd]: http status code OK 13:45:15 DEBUG --- status code --- 13:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:15 DEBUG --- status code --- 13:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:15 DEBUG --- http response --- 13:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:15 DEBUG --- http response --- 13:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:15 DEBUG --- http response --- 13:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911915" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG --- http response --- 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "pod": "ds-cts-0" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "pod": "ds-idrepo-0" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "0.003011840794709845" 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911916" 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "pod": "ds-cts-0" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "pod": "ds-idrepo-0" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "0" 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "instance": "10.142.0.11:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "instance": "10.142.0.51:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "0.00046666666666321055" 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 DEBUG "data": { 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "pod": "ds-cts-0" 13:45:16 DEBUG "instance": "10.142.0.11:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "pod": "ds-idrepo-0" 13:45:16 DEBUG "instance": "10.142.0.51:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "423.0551566080978" 13:45:16 DEBUG "73837.13333333333" 13:45:16 DEBUG ] 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "instance": "10.142.0.11:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "instance": "10.142.0.51:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "92926.00000000001" 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "instance": "10.142.0.11:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "instance": "10.142.0.51:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "18841.6" 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "0.04885756322211705" 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "instance": "10.142.0.11:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "instance": "10.142.0.51:9100" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "0.0003333333333330302" 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911916" 13:45:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911916" 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "pod": "ds-idrepo-0" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "758779904" 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:16 INFO [http_cmd]: http status code OK 13:45:16 DEBUG --- status code --- 13:45:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:45:16 DEBUG --- http response --- 13:45:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:45:16 DEBUG ----- output ----- 13:45:16 DEBUG { 13:45:16 DEBUG "status": "success", 13:45:16 DEBUG "data": { 13:45:16 DEBUG "resultType": "vector", 13:45:16 DEBUG "result": [ 13:45:16 DEBUG { 13:45:16 DEBUG "metric": { 13:45:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG [...] 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG 13:45:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:45:16 DEBUG }, 13:45:16 DEBUG "value": [ 13:45:16 DEBUG 1735911916, 13:45:16 DEBUG "14993223680" 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG ] 13:45:16 DEBUG } 13:45:16 DEBUG } 13:45:27 INFO 13:45:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:45:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:45:27 INFO [loop_until]: OK (rc = 0) 13:45:27 DEBUG --- stdout --- 13:45:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 112m 0% 3850Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 80m 0% 19057Mi 67% 13:45:27 DEBUG --- stderr --- 13:45:27 DEBUG 13:45:30 INFO 13:45:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:45:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:45:30 INFO [loop_until]: OK (rc = 0) 13:45:30 DEBUG --- stdout --- 13:45:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:45:30 DEBUG --- stderr --- 13:45:30 DEBUG 13:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735911975" 13:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911975" 13:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911975" 13:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911975" 13:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911975" 13:46:15 ERROR [http_cmd]: ERROR 13:46:15 ERROR --- status code --- 13:46:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:46:15 ERROR --- http response --- 13:46:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:46:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:46:15 INFO [http_cmd]: http status code OK 13:46:15 DEBUG --- status code --- 13:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:15 DEBUG --- http response --- 13:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:15 INFO [http_cmd]: http status code OK 13:46:15 DEBUG --- status code --- 13:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:15 INFO [http_cmd]: http status code OK 13:46:15 DEBUG --- http response --- 13:46:15 INFO [http_cmd]: http status code OK 13:46:15 DEBUG --- status code --- 13:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:15 DEBUG --- status code --- 13:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:15 DEBUG --- http response --- 13:46:15 DEBUG --- http response --- 13:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "pod": "ds-idrepo-0" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "0.004127234857007754" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "pod": "ds-idrepo-0" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "0" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "instance": "10.142.0.11:9100" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "instance": "10.142.0.51:9100" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "0.0003999999999981204" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:46:16 DEBUG [...] 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG 13:46:16 DEBUG { 13:46:16 DEBUG 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG 13:46:16 DEBUG "data": { 13:46:16 DEBUG "pod": "ds-idrepo-0" 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG }, 13:46:16 DEBUG "result": [ 13:46:16 DEBUG "value": [ 13:46:16 DEBUG { 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "825.9107396807225" 13:46:16 DEBUG "instance": "10.142.0.11:9100" 13:46:16 DEBUG ] 13:46:16 DEBUG }, 13:46:16 DEBUG } 13:46:16 DEBUG "value": [ 13:46:16 DEBUG ] 13:46:16 DEBUG 13:46:16 DEBUG } 13:46:16 DEBUG 13:46:16 DEBUG } 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "instance": "10.142.0.51:9100" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "75378.13333333335" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "instance": "10.142.0.11:9100" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "instance": "10.142.0.51:9100" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "93862.6" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911976" 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "instance": "10.142.0.11:9100" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "instance": "10.142.0.51:9100" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "18158.93333333333" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "0.03903584902425663" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "instance": "10.142.0.11:9100" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "instance": "10.142.0.51:9100" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "0.0006666666666660604" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735911976" 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735911976" 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "pod": "ds-idrepo-0" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "758779904" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:16 INFO [http_cmd]: http status code OK 13:46:16 DEBUG --- status code --- 13:46:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:46:16 DEBUG --- http response --- 13:46:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:46:16 DEBUG ----- output ----- 13:46:16 DEBUG { 13:46:16 DEBUG "status": "success", 13:46:16 DEBUG "data": { 13:46:16 DEBUG "resultType": "vector", 13:46:16 DEBUG "result": [ 13:46:16 DEBUG { 13:46:16 DEBUG "metric": { 13:46:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG [...] 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG 13:46:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:46:16 DEBUG }, 13:46:16 DEBUG "value": [ 13:46:16 DEBUG 1735911976, 13:46:16 DEBUG "14987407360" 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG ] 13:46:16 DEBUG } 13:46:16 DEBUG } 13:46:27 INFO 13:46:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:46:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:46:27 INFO [loop_until]: OK (rc = 0) 13:46:27 DEBUG --- stdout --- 13:46:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 112m 0% 3850Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 80m 0% 19057Mi 67% 13:46:27 DEBUG --- stderr --- 13:46:27 DEBUG 13:46:30 INFO 13:46:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:46:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:46:30 INFO [loop_until]: OK (rc = 0) 13:46:30 DEBUG --- stdout --- 13:46:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:46:30 DEBUG --- stderr --- 13:46:30 DEBUG 13:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912035" 13:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912035" 13:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912035" 13:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912035" 13:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912035" 13:47:15 INFO [http_cmd]: http status code OK 13:47:15 DEBUG --- status code --- 13:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:15 DEBUG --- http response --- 13:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:15 INFO [http_cmd]: http status code OK 13:47:15 DEBUG --- status code --- 13:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:15 DEBUG --- http response --- 13:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:15 ERROR [http_cmd]: ERROR 13:47:15 INFO [http_cmd]: http status code OK 13:47:15 ERROR --- status code --- 13:47:15 DEBUG --- status code --- 13:47:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:15 ERROR --- http response --- 13:47:15 DEBUG --- http response --- 13:47:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:47:15 INFO [http_cmd]: http status code OK 13:47:15 DEBUG --- status code --- 13:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:15 DEBUG --- http response --- 13:47:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG --- http response --- 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- http response --- 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- http response --- 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG { 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "data": { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG { 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "pod": "ds-cts-0" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG "0.0016028756289360971" 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG { 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "data": { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG { 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "pod": "ds-cts-0" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG "0" 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG { 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "data": { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG { 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "instance": "10.142.0.11:9100" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "instance": "10.142.0.51:9100" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG "0.0012666666666670305" 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG { 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG "data": { 13:47:16 DEBUG { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG "data": { 13:47:16 DEBUG { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "result": [ 13:47:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:47:16 DEBUG { 13:47:16 DEBUG }, 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "value": [ 13:47:16 DEBUG "instance": "10.142.0.11:9100" 13:47:16 DEBUG 13:47:16 DEBUG }, 13:47:16 DEBUG 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "pod": "ds-cts-0" 13:47:16 DEBUG 13:47:16 DEBUG }, 13:47:16 DEBUG 13:47:16 DEBUG "value": [ 13:47:16 DEBUG "instance": "10.142.0.51:9100" 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG }, 13:47:16 DEBUG "415.0508795916507" 13:47:16 DEBUG "value": [ 13:47:16 DEBUG ] 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG } 13:47:16 DEBUG "73610.79999999999" 13:47:16 DEBUG ] 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG { 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "data": { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG { 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "instance": "10.142.0.11:9100" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "instance": "10.142.0.51:9100" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG "92683.76666666666" 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912036" 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG { 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "data": { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG { 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "instance": "10.142.0.11:9100" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "instance": "10.142.0.51:9100" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG "24166.399999999998" 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG { 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "data": { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG { 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG "0.03880672522074472" 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG { 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "data": { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG { 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "instance": "10.142.0.11:9100" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "instance": "10.142.0.51:9100" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG "0.0013333333333335417" 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG { 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "data": { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG { 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "pod": "ds-idrepo-0" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG "758779904" 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912036" 13:47:16 INFO [http_cmd]: http status code OK 13:47:16 DEBUG --- status code --- 13:47:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:47:16 DEBUG --- http response --- 13:47:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:47:16 DEBUG ----- output ----- 13:47:16 DEBUG { 13:47:16 DEBUG "status": "success", 13:47:16 DEBUG "data": { 13:47:16 DEBUG "resultType": "vector", 13:47:16 DEBUG "result": [ 13:47:16 DEBUG { 13:47:16 DEBUG "metric": { 13:47:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG [...] 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG 13:47:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:47:16 DEBUG }, 13:47:16 DEBUG "value": [ 13:47:16 DEBUG 1735912036, 13:47:16 DEBUG "14987800576" 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG ] 13:47:16 DEBUG } 13:47:16 DEBUG } 13:47:27 INFO 13:47:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:47:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:47:27 INFO [loop_until]: OK (rc = 0) 13:47:27 DEBUG --- stdout --- 13:47:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 110m 0% 3863Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 77m 0% 19058Mi 67% 13:47:27 DEBUG --- stderr --- 13:47:27 DEBUG 13:47:30 INFO 13:47:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:47:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:47:30 INFO [loop_until]: OK (rc = 0) 13:47:30 DEBUG --- stdout --- 13:47:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:47:30 DEBUG --- stderr --- 13:47:30 DEBUG 13:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912095" 13:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912095" 13:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912095" 13:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912095" 13:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912095" 13:48:15 INFO [http_cmd]: http status code OK 13:48:15 INFO [http_cmd]: http status code OK 13:48:15 DEBUG --- status code --- 13:48:15 INFO [http_cmd]: http status code OK 13:48:15 DEBUG --- status code --- 13:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:15 DEBUG --- status code --- 13:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:15 DEBUG --- http response --- 13:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:15 DEBUG --- http response --- 13:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:15 DEBUG --- http response --- 13:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:15 INFO [http_cmd]: http status code OK 13:48:15 DEBUG --- status code --- 13:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:15 DEBUG --- http response --- 13:48:15 ERROR [http_cmd]: ERROR 13:48:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:15 ERROR --- status code --- 13:48:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:48:15 ERROR --- http response --- 13:48:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:48:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- status code --- 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "0.0006501368539454109" 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912096" 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "0" 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "instance": "10.142.0.11:9100" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "instance": "10.142.0.51:9100" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "0.00039999999999906777" 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "instance": "10.142.0.11:9100" 13:48:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:48:16 DEBUG }, 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "instance": "10.142.0.51:9100" 13:48:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:48:16 DEBUG }, 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "74900.63333333332" 13:48:16 DEBUG "395.6850748671712" 13:48:16 DEBUG ] 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "instance": "10.142.0.11:9100" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "instance": "10.142.0.51:9100" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "94408.16666666666" 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912096" 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "instance": "10.142.0.11:9100" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "instance": "10.142.0.51:9100" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "19387.733333333334" 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "0.039095628831617275" 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "instance": "10.142.0.11:9100" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "instance": "10.142.0.51:9100" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "0.0003333333333330302" 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "pod": "ds-idrepo-0" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "758779904" 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912096" 13:48:16 INFO [http_cmd]: http status code OK 13:48:16 DEBUG --- status code --- 13:48:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:48:16 DEBUG --- http response --- 13:48:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:48:16 DEBUG ----- output ----- 13:48:16 DEBUG { 13:48:16 DEBUG "status": "success", 13:48:16 DEBUG "data": { 13:48:16 DEBUG "resultType": "vector", 13:48:16 DEBUG "result": [ 13:48:16 DEBUG { 13:48:16 DEBUG "metric": { 13:48:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG [...] 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG 13:48:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:48:16 DEBUG }, 13:48:16 DEBUG "value": [ 13:48:16 DEBUG 1735912096, 13:48:16 DEBUG "14990389248" 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG ] 13:48:16 DEBUG } 13:48:16 DEBUG } 13:48:27 INFO 13:48:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:48:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:48:27 INFO [loop_until]: OK (rc = 0) 13:48:27 DEBUG --- stdout --- 13:48:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 109m 0% 3855Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 77m 0% 19061Mi 67% 13:48:27 DEBUG --- stderr --- 13:48:27 DEBUG 13:48:30 INFO 13:48:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:48:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:48:30 INFO [loop_until]: OK (rc = 0) 13:48:30 DEBUG --- stdout --- 13:48:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:48:30 DEBUG --- stderr --- 13:48:30 DEBUG 13:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912155" 13:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912155" 13:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912155" 13:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912155" 13:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912155" 13:49:15 INFO [http_cmd]: http status code OK 13:49:15 DEBUG --- status code --- 13:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:15 DEBUG --- http response --- 13:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:15 INFO [http_cmd]: http status code OK 13:49:15 DEBUG --- status code --- 13:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:15 DEBUG --- http response --- 13:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:15 INFO [http_cmd]: http status code OK 13:49:15 DEBUG --- status code --- 13:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:15 DEBUG --- http response --- 13:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:15 ERROR [http_cmd]: ERROR 13:49:15 ERROR --- status code --- 13:49:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:49:15 ERROR --- http response --- 13:49:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:49:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:49:15 INFO [http_cmd]: http status code OK 13:49:15 DEBUG --- status code --- 13:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:15 DEBUG --- http response --- 13:49:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- http response --- 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG --- status code --- 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG --- status code --- 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG { 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "data": { 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "result": [ 13:49:16 DEBUG { 13:49:16 DEBUG "metric": { 13:49:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG "pod": "ds-cts-0" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG "0.0028024473000321307" 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG { 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "data": { 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "result": [ 13:49:16 DEBUG { 13:49:16 DEBUG "metric": { 13:49:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG "pod": "ds-cts-0" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG "0" 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG { 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "data": { 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "result": [ 13:49:16 DEBUG { 13:49:16 DEBUG "metric": { 13:49:16 DEBUG "instance": "10.142.0.11:9100" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG "instance": "10.142.0.51:9100" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG "0.0005333333333320904" 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:49:16 DEBUG { 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG { 13:49:16 DEBUG "data": { 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "data": { 13:49:16 DEBUG "result": [ 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG { 13:49:16 DEBUG "result": [ 13:49:16 DEBUG "metric": { 13:49:16 DEBUG { 13:49:16 DEBUG "instance": "10.142.0.11:9100" 13:49:16 DEBUG "metric": { 13:49:16 DEBUG }, 13:49:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:49:16 DEBUG "value": [ 13:49:16 DEBUG }, 13:49:16 DEBUG 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG "instance": "10.142.0.51:9100" 13:49:16 DEBUG }, 13:49:16 DEBUG 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG 13:49:16 DEBUG "71848" 13:49:16 DEBUG "pod": "ds-cts-0" 13:49:16 DEBUG ] 13:49:16 DEBUG }, 13:49:16 DEBUG } 13:49:16 DEBUG "value": [ 13:49:16 DEBUG ] 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG } 13:49:16 DEBUG "0" 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG { 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "data": { 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "result": [ 13:49:16 DEBUG { 13:49:16 DEBUG "metric": { 13:49:16 DEBUG "instance": "10.142.0.11:9100" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG "instance": "10.142.0.51:9100" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG "89081.26666666666" 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912156" 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG { 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "data": { 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "result": [ 13:49:16 DEBUG { 13:49:16 DEBUG "metric": { 13:49:16 DEBUG "instance": "10.142.0.11:9100" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG "instance": "10.142.0.51:9100" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG "21299.2" 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG { 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "data": { 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "result": [ 13:49:16 DEBUG { 13:49:16 DEBUG "metric": { 13:49:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG "0.04345106020746055" 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG { 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "data": { 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "result": [ 13:49:16 DEBUG { 13:49:16 DEBUG "metric": { 13:49:16 DEBUG "instance": "10.142.0.11:9100" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG "instance": "10.142.0.51:9100" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG "0.0009999999999990906" 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912156" 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG { 13:49:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912156" 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "data": { 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "result": [ 13:49:16 DEBUG { 13:49:16 DEBUG "metric": { 13:49:16 DEBUG "pod": "ds-idrepo-0" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG "758779904" 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:16 INFO [http_cmd]: http status code OK 13:49:16 DEBUG --- status code --- 13:49:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:49:16 DEBUG --- http response --- 13:49:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:49:16 DEBUG ----- output ----- 13:49:16 DEBUG { 13:49:16 DEBUG "status": "success", 13:49:16 DEBUG "data": { 13:49:16 DEBUG "resultType": "vector", 13:49:16 DEBUG "result": [ 13:49:16 DEBUG { 13:49:16 DEBUG "metric": { 13:49:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG [...] 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG 13:49:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:49:16 DEBUG }, 13:49:16 DEBUG "value": [ 13:49:16 DEBUG 1735912156, 13:49:16 DEBUG "14989639680" 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG ] 13:49:16 DEBUG } 13:49:16 DEBUG } 13:49:27 INFO 13:49:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:49:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:49:27 INFO [loop_until]: OK (rc = 0) 13:49:27 DEBUG --- stdout --- 13:49:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3854Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 79m 0% 19061Mi 67% 13:49:27 DEBUG --- stderr --- 13:49:27 DEBUG 13:49:30 INFO 13:49:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:49:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:49:30 INFO [loop_until]: OK (rc = 0) 13:49:30 DEBUG --- stdout --- 13:49:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:49:30 DEBUG --- stderr --- 13:49:30 DEBUG 13:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912215" 13:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912215" 13:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912215" 13:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912215" 13:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912215" 13:50:15 INFO [http_cmd]: http status code OK 13:50:15 DEBUG --- status code --- 13:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:15 DEBUG --- http response --- 13:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:15 INFO [http_cmd]: http status code OK 13:50:15 DEBUG --- status code --- 13:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:15 DEBUG --- http response --- 13:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:15 INFO [http_cmd]: http status code OK 13:50:15 DEBUG --- status code --- 13:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:15 DEBUG --- http response --- 13:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:15 ERROR [http_cmd]: ERROR 13:50:15 ERROR --- status code --- 13:50:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:50:15 ERROR --- http response --- 13:50:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:50:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:50:15 INFO [http_cmd]: http status code OK 13:50:15 DEBUG --- status code --- 13:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:15 DEBUG --- http response --- 13:50:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912216" 13:50:16 DEBUG { 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "data": { 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "result": [ 13:50:16 DEBUG { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "pod": "ds-cts-0" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG [...] 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "0.0011225993449895315" 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG { 13:50:16 DEBUG { 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "data": { 13:50:16 DEBUG "data": { 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "result": [ 13:50:16 DEBUG "result": [ 13:50:16 DEBUG { 13:50:16 DEBUG { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "pod": "ds-cts-0" 13:50:16 DEBUG "instance": "10.142.0.11:9100" 13:50:16 DEBUG }, 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG [...] 13:50:16 DEBUG [...] 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:50:16 DEBUG "instance": "10.142.0.51:9100" 13:50:16 DEBUG }, 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "0" 13:50:16 DEBUG "0.00046666666666321055" 13:50:16 DEBUG ] 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 DEBUG ] 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG { 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "data": { 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "result": [ 13:50:16 DEBUG { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "instance": "10.142.0.11:9100" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG [...] 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:50:16 DEBUG 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG 13:50:16 DEBUG { 13:50:16 DEBUG "instance": "10.142.0.51:9100" 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG }, 13:50:16 DEBUG "data": { 13:50:16 DEBUG "value": [ 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "result": [ 13:50:16 DEBUG "74610.96666666669" 13:50:16 DEBUG { 13:50:16 DEBUG ] 13:50:16 DEBUG "metric": { 13:50:16 DEBUG } 13:50:16 DEBUG "pod": "ds-cts-0" 13:50:16 DEBUG ] 13:50:16 DEBUG }, 13:50:16 DEBUG } 13:50:16 DEBUG "value": [ 13:50:16 DEBUG } 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG [...] 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "0" 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG { 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "data": { 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "result": [ 13:50:16 DEBUG { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "instance": "10.142.0.11:9100" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG [...] 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG "instance": "10.142.0.51:9100" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "93648.36666666668" 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG { 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "data": { 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "result": [ 13:50:16 DEBUG { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "instance": "10.142.0.11:9100" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG [...] 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG "instance": "10.142.0.51:9100" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "32358.399999999998" 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG { 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "data": { 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "result": [ 13:50:16 DEBUG { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG [...] 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "0.04527210291239421" 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG { 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "data": { 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "result": [ 13:50:16 DEBUG { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "instance": "10.142.0.11:9100" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG [...] 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG "instance": "10.142.0.51:9100" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "0" 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912216" 13:50:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912216" 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG { 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "data": { 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "result": [ 13:50:16 DEBUG { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "pod": "ds-idrepo-0" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG [...] 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "758779904" 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:16 INFO [http_cmd]: http status code OK 13:50:16 DEBUG --- status code --- 13:50:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:50:16 DEBUG --- http response --- 13:50:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:50:16 DEBUG ----- output ----- 13:50:16 DEBUG { 13:50:16 DEBUG "status": "success", 13:50:16 DEBUG "data": { 13:50:16 DEBUG "resultType": "vector", 13:50:16 DEBUG "result": [ 13:50:16 DEBUG { 13:50:16 DEBUG "metric": { 13:50:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG [...] 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG 13:50:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:50:16 DEBUG }, 13:50:16 DEBUG "value": [ 13:50:16 DEBUG 1735912216, 13:50:16 DEBUG "14990209024" 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG ] 13:50:16 DEBUG } 13:50:16 DEBUG } 13:50:27 INFO 13:50:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:50:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:50:27 INFO [loop_until]: OK (rc = 0) 13:50:27 DEBUG --- stdout --- 13:50:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 113m 0% 3865Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 74m 0% 19055Mi 67% 13:50:27 DEBUG --- stderr --- 13:50:27 DEBUG 13:50:30 INFO 13:50:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:50:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:50:30 INFO [loop_until]: OK (rc = 0) 13:50:30 DEBUG --- stdout --- 13:50:30 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 2m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:50:30 DEBUG --- stderr --- 13:50:30 DEBUG 13:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912275" 13:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912275" 13:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912275" 13:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912275" 13:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912275" 13:51:15 INFO [http_cmd]: http status code OK 13:51:15 DEBUG --- status code --- 13:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:15 DEBUG --- http response --- 13:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:15 INFO [http_cmd]: http status code OK 13:51:15 DEBUG --- status code --- 13:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:15 DEBUG --- http response --- 13:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:15 ERROR [http_cmd]: ERROR 13:51:15 ERROR --- status code --- 13:51:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:51:15 ERROR --- http response --- 13:51:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:51:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:51:15 INFO [http_cmd]: http status code OK 13:51:15 DEBUG --- status code --- 13:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:15 DEBUG --- http response --- 13:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:15 INFO [http_cmd]: http status code OK 13:51:15 DEBUG --- status code --- 13:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:15 DEBUG --- http response --- 13:51:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 DEBUG --- http response --- 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG { 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "data": { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "result": [ 13:51:16 DEBUG { 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG "pod": "ds-cts-0" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG "0.002734149572186156" 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG { 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG "data": { 13:51:16 DEBUG { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "result": [ 13:51:16 DEBUG "data": { 13:51:16 DEBUG { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "result": [ 13:51:16 DEBUG "instance": "10.142.0.11:9100" 13:51:16 DEBUG { 13:51:16 DEBUG }, 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "value": [ 13:51:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:51:16 DEBUG 13:51:16 DEBUG }, 13:51:16 DEBUG 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG "instance": "10.142.0.51:9100" 13:51:16 DEBUG 13:51:16 DEBUG }, 13:51:16 DEBUG 13:51:16 DEBUG "value": [ 13:51:16 DEBUG "pod": "ds-cts-0" 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG }, 13:51:16 DEBUG "0.0003999999999981204" 13:51:16 DEBUG "value": [ 13:51:16 DEBUG ] 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG } 13:51:16 DEBUG "0" 13:51:16 DEBUG ] 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG { 13:51:16 DEBUG { 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "data": { 13:51:16 DEBUG "data": { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "result": [ 13:51:16 DEBUG "result": [ 13:51:16 DEBUG { 13:51:16 DEBUG { 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:51:16 DEBUG "instance": "10.142.0.11:9100" 13:51:16 DEBUG }, 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG "pod": "ds-cts-0" 13:51:16 DEBUG "instance": "10.142.0.51:9100" 13:51:16 DEBUG }, 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG "733.1829735438371" 13:51:16 DEBUG "74929.90000000001" 13:51:16 DEBUG ] 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 DEBUG ] 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG { 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "data": { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "result": [ 13:51:16 DEBUG { 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "instance": "10.142.0.11:9100" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG "instance": "10.142.0.51:9100" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG "94237.59999999999" 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG { 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "data": { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "result": [ 13:51:16 DEBUG { 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "instance": "10.142.0.11:9100" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG "instance": "10.142.0.51:9100" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG "22801.066666666666" 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG { 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "data": { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "result": [ 13:51:16 DEBUG { 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG "0.043157134076592885" 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG { 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "data": { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "result": [ 13:51:16 DEBUG { 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "instance": "10.142.0.11:9100" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG "instance": "10.142.0.51:9100" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG "0" 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912276" 13:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912276" 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG { 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "data": { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "result": [ 13:51:16 DEBUG { 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "pod": "ds-idrepo-0" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG "758779904" 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:16 INFO [http_cmd]: http status code OK 13:51:16 DEBUG --- status code --- 13:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:51:16 DEBUG --- http response --- 13:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:51:16 DEBUG ----- output ----- 13:51:16 DEBUG { 13:51:16 DEBUG "status": "success", 13:51:16 DEBUG "data": { 13:51:16 DEBUG "resultType": "vector", 13:51:16 DEBUG "result": [ 13:51:16 DEBUG { 13:51:16 DEBUG "metric": { 13:51:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG [...] 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG 13:51:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:51:16 DEBUG }, 13:51:16 DEBUG "value": [ 13:51:16 DEBUG 1735912276, 13:51:16 DEBUG "14988587008" 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG ] 13:51:16 DEBUG } 13:51:16 DEBUG } 13:51:27 INFO 13:51:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:51:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:51:27 INFO [loop_until]: OK (rc = 0) 13:51:27 DEBUG --- stdout --- 13:51:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3880Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 80m 0% 19058Mi 67% 13:51:27 DEBUG --- stderr --- 13:51:27 DEBUG 13:51:30 INFO 13:51:30 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:51:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:51:31 INFO [loop_until]: OK (rc = 0) 13:51:31 DEBUG --- stdout --- 13:51:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:51:31 DEBUG --- stderr --- 13:51:31 DEBUG 13:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912335" 13:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912335" 13:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912335" 13:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912335" 13:52:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912335" 13:52:15 INFO [http_cmd]: http status code OK 13:52:15 INFO [http_cmd]: http status code OK 13:52:15 INFO [http_cmd]: http status code OK 13:52:15 DEBUG --- status code --- 13:52:15 ERROR [http_cmd]: ERROR 13:52:15 INFO [http_cmd]: http status code OK 13:52:15 DEBUG --- status code --- 13:52:15 DEBUG --- status code --- 13:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:15 ERROR --- status code --- 13:52:15 DEBUG --- status code --- 13:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:15 DEBUG --- http response --- 13:52:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:52:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:15 DEBUG --- http response --- 13:52:15 DEBUG --- http response --- 13:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:15 ERROR --- http response --- 13:52:15 DEBUG --- http response --- 13:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:52:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG { 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "data": { 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "result": [ 13:52:16 DEBUG { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "pod": "ds-cts-0" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG [...] 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG "0.0009556283967461504" 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG ] 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG } 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG } 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG { 13:52:16 DEBUG { 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "data": { 13:52:16 DEBUG "data": { 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "result": [ 13:52:16 DEBUG "result": [ 13:52:16 DEBUG { 13:52:16 DEBUG { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "instance": "10.142.0.11:9100" 13:52:16 DEBUG "pod": "ds-cts-0" 13:52:16 DEBUG }, 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG [...] 13:52:16 DEBUG [...] 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG "instance": "10.142.0.51:9100" 13:52:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:52:16 DEBUG }, 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG "0.0010666666666679701" 13:52:16 DEBUG "0" 13:52:16 DEBUG ] 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 DEBUG ] 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG { 13:52:16 DEBUG { 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "data": { 13:52:16 DEBUG "data": { 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "result": [ 13:52:16 DEBUG "result": [ 13:52:16 DEBUG { 13:52:16 DEBUG { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "instance": "10.142.0.11:9100" 13:52:16 DEBUG "pod": "ds-cts-0" 13:52:16 DEBUG }, 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG [...] 13:52:16 DEBUG [...] 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG "instance": "10.142.0.51:9100" 13:52:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:52:16 DEBUG }, 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG "74724.06666666667" 13:52:16 DEBUG "759.7091823489063" 13:52:16 DEBUG ] 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 DEBUG ] 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG { 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "data": { 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "result": [ 13:52:16 DEBUG { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "instance": "10.142.0.11:9100" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG [...] 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG "instance": "10.142.0.51:9100" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG "94349.90000000001" 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG { 13:52:16 DEBUG { 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "data": { 13:52:16 DEBUG "data": { 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "result": [ 13:52:16 DEBUG "result": [ 13:52:16 DEBUG { 13:52:16 DEBUG { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:52:16 DEBUG "instance": "10.142.0.11:9100" 13:52:16 DEBUG }, 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG [...] 13:52:16 DEBUG [...] 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:52:16 DEBUG "instance": "10.142.0.51:9100" 13:52:16 DEBUG }, 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG "0.040841893760019926" 13:52:16 DEBUG "25258.666666666664" 13:52:16 DEBUG ] 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 DEBUG ] 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG { 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "data": { 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "result": [ 13:52:16 DEBUG { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "instance": "10.142.0.11:9100" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG [...] 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG "instance": "10.142.0.51:9100" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG "0.0013333333333340155" 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912336" 13:52:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912336" 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG { 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "data": { 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "result": [ 13:52:16 DEBUG { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "pod": "ds-idrepo-0" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG [...] 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG "758779904" 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:16 INFO [http_cmd]: http status code OK 13:52:16 DEBUG --- status code --- 13:52:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:52:16 DEBUG --- http response --- 13:52:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:52:16 DEBUG ----- output ----- 13:52:16 DEBUG { 13:52:16 DEBUG "status": "success", 13:52:16 DEBUG "data": { 13:52:16 DEBUG "resultType": "vector", 13:52:16 DEBUG "result": [ 13:52:16 DEBUG { 13:52:16 DEBUG "metric": { 13:52:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG [...] 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG 13:52:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:52:16 DEBUG }, 13:52:16 DEBUG "value": [ 13:52:16 DEBUG 1735912336, 13:52:16 DEBUG "14988013568" 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG ] 13:52:16 DEBUG } 13:52:16 DEBUG } 13:52:27 INFO 13:52:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:52:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:52:27 INFO [loop_until]: OK (rc = 0) 13:52:27 DEBUG --- stdout --- 13:52:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 116m 0% 3846Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 76m 0% 19058Mi 67% 13:52:27 DEBUG --- stderr --- 13:52:27 DEBUG 13:52:31 INFO 13:52:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:52:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:52:31 INFO [loop_until]: OK (rc = 0) 13:52:31 DEBUG --- stdout --- 13:52:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:52:31 DEBUG --- stderr --- 13:52:31 DEBUG 13:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912395" 13:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912395" 13:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912395" 13:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912395" 13:53:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912395" 13:53:15 INFO [http_cmd]: http status code OK 13:53:15 INFO [http_cmd]: http status code OK 13:53:15 INFO [http_cmd]: http status code OK 13:53:15 DEBUG --- status code --- 13:53:15 ERROR [http_cmd]: ERROR 13:53:15 DEBUG --- status code --- 13:53:15 INFO [http_cmd]: http status code OK 13:53:15 DEBUG --- status code --- 13:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:15 ERROR --- status code --- 13:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:15 DEBUG --- http response --- 13:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:15 DEBUG --- http response --- 13:53:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:53:15 DEBUG --- status code --- 13:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:15 DEBUG --- http response --- 13:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:15 ERROR --- http response --- 13:53:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:53:15 DEBUG --- http response --- 13:53:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:53:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG --- status code --- 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "pod": "ds-cts-0" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "0.002745448783409814" 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "instance": "10.142.0.11:9100" 13:53:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:53:16 DEBUG }, 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "instance": "10.142.0.51:9100" 13:53:16 DEBUG "pod": "ds-cts-0" 13:53:16 DEBUG }, 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "0.00039999999999812034" 13:53:16 DEBUG "0" 13:53:16 DEBUG ] 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "pod": "ds-cts-0" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "432.69129194689947" 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "instance": "10.142.0.11:9100" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "instance": "10.142.0.51:9100" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "74036.66666666666" 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "instance": "10.142.0.11:9100" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "instance": "10.142.0.51:9100" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "93314.03333333333" 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "instance": "10.142.0.11:9100" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "instance": "10.142.0.51:9100" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "25122.13333333333" 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "0.04005799030448516" 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "instance": "10.142.0.11:9100" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "instance": "10.142.0.51:9100" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "0.0003333333333330302" 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912396" 13:53:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912396" 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "pod": "ds-idrepo-0" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "758779904" 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:16 INFO [http_cmd]: http status code OK 13:53:16 DEBUG --- status code --- 13:53:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:53:16 DEBUG --- http response --- 13:53:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:53:16 DEBUG ----- output ----- 13:53:16 DEBUG { 13:53:16 DEBUG "status": "success", 13:53:16 DEBUG "data": { 13:53:16 DEBUG "resultType": "vector", 13:53:16 DEBUG "result": [ 13:53:16 DEBUG { 13:53:16 DEBUG "metric": { 13:53:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG [...] 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG 13:53:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:53:16 DEBUG }, 13:53:16 DEBUG "value": [ 13:53:16 DEBUG 1735912396, 13:53:16 DEBUG "14988861440" 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG ] 13:53:16 DEBUG } 13:53:16 DEBUG } 13:53:27 INFO 13:53:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:53:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:53:27 INFO [loop_until]: OK (rc = 0) 13:53:27 DEBUG --- stdout --- 13:53:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 119m 0% 3862Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 80m 0% 19058Mi 67% 13:53:27 DEBUG --- stderr --- 13:53:27 DEBUG 13:53:31 INFO 13:53:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:53:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:53:31 INFO [loop_until]: OK (rc = 0) 13:53:31 DEBUG --- stdout --- 13:53:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:53:31 DEBUG --- stderr --- 13:53:31 DEBUG 13:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912455" 13:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912455" 13:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912455" 13:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912455" 13:54:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912455" 13:54:15 INFO [http_cmd]: http status code OK 13:54:15 INFO [http_cmd]: http status code OK 13:54:15 INFO [http_cmd]: http status code OK 13:54:15 DEBUG --- status code --- 13:54:15 DEBUG --- status code --- 13:54:15 DEBUG --- status code --- 13:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:15 DEBUG --- http response --- 13:54:15 DEBUG --- http response --- 13:54:15 DEBUG --- http response --- 13:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:15 ERROR [http_cmd]: ERROR 13:54:15 INFO [http_cmd]: http status code OK 13:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:15 ERROR --- status code --- 13:54:15 DEBUG --- status code --- 13:54:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:54:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:15 ERROR --- http response --- 13:54:15 DEBUG --- http response --- 13:54:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:54:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- status code --- 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG { 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "data": { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "result": [ 13:54:16 DEBUG { 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG "pod": "ds-cts-0" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG "0.001945565997464036" 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG { 13:54:16 DEBUG { 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "data": { 13:54:16 DEBUG "data": { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "result": [ 13:54:16 DEBUG "result": [ 13:54:16 DEBUG { 13:54:16 DEBUG { 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "instance": "10.142.0.11:9100" 13:54:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:54:16 DEBUG }, 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG "instance": "10.142.0.51:9100" 13:54:16 DEBUG "pod": "ds-cts-0" 13:54:16 DEBUG }, 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG "0.00043333333333066547" 13:54:16 DEBUG "0" 13:54:16 DEBUG ] 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 DEBUG ] 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG { 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "data": { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "result": [ 13:54:16 DEBUG { 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG "pod": "ds-cts-0" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG "415.340707814132" 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG { 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "data": { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "result": [ 13:54:16 DEBUG { 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "instance": "10.142.0.11:9100" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG "instance": "10.142.0.51:9100" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG "81052.03333333334" 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG { 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "data": { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "result": [ 13:54:16 DEBUG { 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "instance": "10.142.0.11:9100" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG "instance": "10.142.0.51:9100" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG "97808.03333333334" 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG { 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG "data": { 13:54:16 DEBUG { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "result": [ 13:54:16 DEBUG "data": { 13:54:16 DEBUG { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "result": [ 13:54:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:54:16 DEBUG { 13:54:16 DEBUG }, 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "value": [ 13:54:16 DEBUG "instance": "10.142.0.11:9100" 13:54:16 DEBUG 13:54:16 DEBUG }, 13:54:16 DEBUG 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:54:16 DEBUG 13:54:16 DEBUG }, 13:54:16 DEBUG 13:54:16 DEBUG "value": [ 13:54:16 DEBUG "instance": "10.142.0.51:9100" 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG }, 13:54:16 DEBUG "0.042122461977385894" 13:54:16 DEBUG "value": [ 13:54:16 DEBUG ] 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG } 13:54:16 DEBUG "23347.199999999997" 13:54:16 DEBUG ] 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG { 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "data": { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "result": [ 13:54:16 DEBUG { 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "instance": "10.142.0.11:9100" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG "instance": "10.142.0.51:9100" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG "0.000666666666666534" 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912456" 13:54:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912456" 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG { 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "data": { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "result": [ 13:54:16 DEBUG { 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "pod": "ds-idrepo-0" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG "758788096" 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:16 INFO [http_cmd]: http status code OK 13:54:16 DEBUG --- status code --- 13:54:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:54:16 DEBUG --- http response --- 13:54:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:54:16 DEBUG ----- output ----- 13:54:16 DEBUG { 13:54:16 DEBUG "status": "success", 13:54:16 DEBUG "data": { 13:54:16 DEBUG "resultType": "vector", 13:54:16 DEBUG "result": [ 13:54:16 DEBUG { 13:54:16 DEBUG "metric": { 13:54:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG [...] 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG 13:54:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:54:16 DEBUG }, 13:54:16 DEBUG "value": [ 13:54:16 DEBUG 1735912456, 13:54:16 DEBUG "14987948032" 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG ] 13:54:16 DEBUG } 13:54:16 DEBUG } 13:54:27 INFO 13:54:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:54:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:54:27 INFO [loop_until]: OK (rc = 0) 13:54:27 DEBUG --- stdout --- 13:54:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 120m 0% 3863Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 80m 0% 19056Mi 67% 13:54:27 DEBUG --- stderr --- 13:54:27 DEBUG 13:54:31 INFO 13:54:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:54:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:54:31 INFO [loop_until]: OK (rc = 0) 13:54:31 DEBUG --- stdout --- 13:54:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:54:31 DEBUG --- stderr --- 13:54:31 DEBUG 13:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912515" 13:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912515" 13:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912515" 13:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912515" 13:55:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912515" 13:55:15 INFO [http_cmd]: http status code OK 13:55:15 DEBUG --- status code --- 13:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:15 DEBUG --- http response --- 13:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:15 ERROR [http_cmd]: ERROR 13:55:15 ERROR --- status code --- 13:55:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:55:15 ERROR --- http response --- 13:55:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:55:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:55:15 INFO [http_cmd]: http status code OK 13:55:15 DEBUG --- status code --- 13:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:15 DEBUG --- http response --- 13:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:15 INFO [http_cmd]: http status code OK 13:55:15 DEBUG --- status code --- 13:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:15 INFO [http_cmd]: http status code OK 13:55:15 DEBUG --- http response --- 13:55:15 DEBUG --- status code --- 13:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:15 DEBUG --- http response --- 13:55:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG { 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG "data": { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "result": [ 13:55:16 DEBUG { 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "pod": "ds-cts-0" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG "0.00040278964773874943" 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG { 13:55:16 DEBUG { 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG "data": { 13:55:16 DEBUG "data": { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "result": [ 13:55:16 DEBUG "result": [ 13:55:16 DEBUG { 13:55:16 DEBUG { 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "instance": "10.142.0.11:9100" 13:55:16 DEBUG "pod": "ds-cts-0" 13:55:16 DEBUG }, 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG "instance": "10.142.0.51:9100" 13:55:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:55:16 DEBUG }, 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG "value": [ 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912516" 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG "0.00046666666666700016" 13:55:16 DEBUG "0" 13:55:16 DEBUG ] 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 DEBUG ] 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG "data": { 13:55:16 DEBUG { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG "result": [ 13:55:16 DEBUG "data": { 13:55:16 DEBUG { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "result": [ 13:55:16 DEBUG "instance": "10.142.0.11:9100" 13:55:16 DEBUG { 13:55:16 DEBUG }, 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "value": [ 13:55:16 DEBUG "pod": "ds-cts-0" 13:55:16 DEBUG 13:55:16 DEBUG }, 13:55:16 DEBUG 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG "instance": "10.142.0.51:9100" 13:55:16 DEBUG 13:55:16 DEBUG }, 13:55:16 DEBUG 13:55:16 DEBUG "value": [ 13:55:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG }, 13:55:16 DEBUG "75017.76666666668" 13:55:16 DEBUG "value": [ 13:55:16 DEBUG ] 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG } 13:55:16 DEBUG "316.8477812828602" 13:55:16 DEBUG ] 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG { 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG "data": { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "result": [ 13:55:16 DEBUG { 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "instance": "10.142.0.11:9100" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG "instance": "10.142.0.51:9100" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG "93683.36666666667" 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG { 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG "data": { 13:55:16 DEBUG { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG "result": [ 13:55:16 DEBUG "data": { 13:55:16 DEBUG { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "result": [ 13:55:16 DEBUG "instance": "10.142.0.11:9100" 13:55:16 DEBUG { 13:55:16 DEBUG }, 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "value": [ 13:55:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:55:16 DEBUG 13:55:16 DEBUG }, 13:55:16 DEBUG 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG "instance": "10.142.0.51:9100" 13:55:16 DEBUG 13:55:16 DEBUG }, 13:55:16 DEBUG 13:55:16 DEBUG "value": [ 13:55:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG }, 13:55:16 DEBUG "23620.266666666666" 13:55:16 DEBUG "value": [ 13:55:16 DEBUG ] 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG } 13:55:16 DEBUG "0.04337340388400817" 13:55:16 DEBUG ] 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG { 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG "data": { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "result": [ 13:55:16 DEBUG { 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "instance": "10.142.0.11:9100" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG "instance": "10.142.0.51:9100" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG "0" 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912516" 13:55:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912516" 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG { 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG "data": { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "result": [ 13:55:16 DEBUG { 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "pod": "ds-idrepo-0" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG "758779904" 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:16 INFO [http_cmd]: http status code OK 13:55:16 DEBUG --- status code --- 13:55:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:55:16 DEBUG --- http response --- 13:55:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:55:16 DEBUG ----- output ----- 13:55:16 DEBUG { 13:55:16 DEBUG "status": "success", 13:55:16 DEBUG "data": { 13:55:16 DEBUG "resultType": "vector", 13:55:16 DEBUG "result": [ 13:55:16 DEBUG { 13:55:16 DEBUG "metric": { 13:55:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG [...] 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG 13:55:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:55:16 DEBUG }, 13:55:16 DEBUG "value": [ 13:55:16 DEBUG 1735912516, 13:55:16 DEBUG "14985256960" 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG ] 13:55:16 DEBUG } 13:55:16 DEBUG } 13:55:27 INFO 13:55:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:55:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:55:27 INFO [loop_until]: OK (rc = 0) 13:55:27 DEBUG --- stdout --- 13:55:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 116m 0% 3811Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 74m 0% 19056Mi 67% 13:55:27 DEBUG --- stderr --- 13:55:27 DEBUG 13:55:31 INFO 13:55:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:55:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:55:31 INFO [loop_until]: OK (rc = 0) 13:55:31 DEBUG --- stdout --- 13:55:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:55:31 DEBUG --- stderr --- 13:55:31 DEBUG 13:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912575" 13:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912575" 13:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912575" 13:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912575" 13:56:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912575" 13:56:15 ERROR [http_cmd]: ERROR 13:56:15 ERROR --- status code --- 13:56:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:56:15 ERROR --- http response --- 13:56:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:56:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:56:15 INFO [http_cmd]: http status code OK 13:56:15 DEBUG --- status code --- 13:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:15 DEBUG --- http response --- 13:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:15 INFO [http_cmd]: http status code OK 13:56:15 INFO [http_cmd]: http status code OK 13:56:15 DEBUG --- status code --- 13:56:15 DEBUG --- status code --- 13:56:15 INFO [http_cmd]: http status code OK 13:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:15 DEBUG --- status code --- 13:56:15 DEBUG --- http response --- 13:56:15 DEBUG --- http response --- 13:56:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:15 DEBUG --- http response --- 13:56:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "result": [ 13:56:16 DEBUG { 13:56:16 DEBUG "metric": { 13:56:16 DEBUG "pod": "ds-cts-0" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "0.0005716759736957748" 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "result": [ 13:56:16 DEBUG { 13:56:16 DEBUG "metric": { 13:56:16 DEBUG "instance": "10.142.0.11:9100" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "instance": "10.142.0.51:9100" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "0.00046666666666794754" 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "result": [ 13:56:16 DEBUG { 13:56:16 DEBUG "metric": { 13:56:16 DEBUG "pod": "ds-cts-0" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "0" 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "result": [ 13:56:16 DEBUG { 13:56:16 DEBUG "metric": { 13:56:16 DEBUG "pod": "ds-cts-0" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "434.29702410405037" 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "result": [ 13:56:16 DEBUG { 13:56:16 DEBUG "metric": { 13:56:16 DEBUG "instance": "10.142.0.11:9100" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "instance": "10.142.0.51:9100" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "77949.26666666669" 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "result": [ 13:56:16 DEBUG { 13:56:16 DEBUG "metric": { 13:56:16 DEBUG "instance": "10.142.0.11:9100" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "instance": "10.142.0.51:9100" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "94216.73333333334" 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:56:16 DEBUG { 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG { 13:56:16 DEBUG "data": { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "result": [ 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG { 13:56:16 DEBUG "result": [ 13:56:16 DEBUG "metric": { 13:56:16 DEBUG { 13:56:16 DEBUG "instance": "10.142.0.11:9100" 13:56:16 DEBUG "metric": { 13:56:16 DEBUG }, 13:56:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:56:16 DEBUG "value": [ 13:56:16 DEBUG }, 13:56:16 DEBUG 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "instance": "10.142.0.51:9100" 13:56:16 DEBUG 13:56:16 DEBUG }, 13:56:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:56:16 DEBUG "value": [ 13:56:16 DEBUG }, 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG "25258.666666666668" 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG ] 13:56:16 DEBUG "0.03946766315968768" 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "result": [ 13:56:16 DEBUG { 13:56:16 DEBUG "metric": { 13:56:16 DEBUG "instance": "10.142.0.11:9100" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "instance": "10.142.0.51:9100" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "0.00033333333333350387" 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912576" 13:56:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912576" 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "result": [ 13:56:16 DEBUG { 13:56:16 DEBUG "metric": { 13:56:16 DEBUG "pod": "ds-idrepo-0" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "758788096" 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:16 INFO [http_cmd]: http status code OK 13:56:16 DEBUG --- status code --- 13:56:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:56:16 DEBUG --- http response --- 13:56:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:56:16 DEBUG ----- output ----- 13:56:16 DEBUG { 13:56:16 DEBUG "status": "success", 13:56:16 DEBUG "data": { 13:56:16 DEBUG "resultType": "vector", 13:56:16 DEBUG "result": [ 13:56:16 DEBUG { 13:56:16 DEBUG "metric": { 13:56:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG [...] 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG 13:56:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:56:16 DEBUG }, 13:56:16 DEBUG "value": [ 13:56:16 DEBUG 1735912576, 13:56:16 DEBUG "14985760768" 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG ] 13:56:16 DEBUG } 13:56:16 DEBUG } 13:56:27 INFO 13:56:27 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:56:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:56:28 INFO [loop_until]: OK (rc = 0) 13:56:28 DEBUG --- stdout --- 13:56:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 116m 0% 3819Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 71m 0% 19059Mi 67% 13:56:28 DEBUG --- stderr --- 13:56:28 DEBUG 13:56:31 INFO 13:56:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:56:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:56:31 INFO [loop_until]: OK (rc = 0) 13:56:31 DEBUG --- stdout --- 13:56:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:56:31 DEBUG --- stderr --- 13:56:31 DEBUG 13:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912635" 13:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912635" 13:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912635" 13:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912635" 13:57:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912635" 13:57:15 INFO [http_cmd]: http status code OK 13:57:15 DEBUG --- status code --- 13:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:15 DEBUG --- http response --- 13:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:15 ERROR [http_cmd]: ERROR 13:57:15 ERROR --- status code --- 13:57:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:57:15 ERROR --- http response --- 13:57:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:57:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:57:15 INFO [http_cmd]: http status code OK 13:57:15 DEBUG --- status code --- 13:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:15 DEBUG --- http response --- 13:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:15 INFO [http_cmd]: http status code OK 13:57:15 INFO [http_cmd]: http status code OK 13:57:15 DEBUG --- status code --- 13:57:15 DEBUG --- status code --- 13:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:15 DEBUG --- http response --- 13:57:15 DEBUG --- http response --- 13:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 DEBUG --- http response --- 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 DEBUG --- status code --- 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG { 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 DEBUG { 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG [...] 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG "pod": "ds-cts-0" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "0" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG { 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 DEBUG { 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "instance": "10.142.0.11:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG [...] 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG "instance": "10.142.0.51:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "0.0014333333333259664" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG { 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 DEBUG { 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG [...] 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "0" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG { 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 DEBUG { 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "instance": "10.142.0.11:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG [...] 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG "instance": "10.142.0.51:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "77209.33333333333" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG { 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 DEBUG { 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG [...] 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "326.3194405594406" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG { 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 DEBUG { 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "instance": "10.142.0.11:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG [...] 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG "instance": "10.142.0.51:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "94080.76666666668" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG { 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG { 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG "metric": { 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG }, 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:57:16 DEBUG "value": [ 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG 13:57:16 DEBUG { 13:57:16 DEBUG 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG 13:57:16 DEBUG "data": { 13:57:16 DEBUG [...] 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG 13:57:16 DEBUG "result": [ 13:57:16 DEBUG 13:57:16 DEBUG { 13:57:16 DEBUG 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:57:16 DEBUG "instance": "10.142.0.11:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG 13:57:16 DEBUG "0.041089365948502814" 13:57:16 DEBUG 13:57:16 DEBUG ] 13:57:16 DEBUG 13:57:16 DEBUG } 13:57:16 DEBUG [...] 13:57:16 DEBUG ] 13:57:16 DEBUG 13:57:16 DEBUG } 13:57:16 DEBUG 13:57:16 DEBUG } 13:57:16 DEBUG 13:57:16 DEBUG "instance": "10.142.0.51:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "27033.6" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG { 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 DEBUG { 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "instance": "10.142.0.11:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG [...] 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG "instance": "10.142.0.51:9100" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "0.0010000000000005115" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912636" 13:57:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912636" 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG { 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 DEBUG { 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "pod": "ds-idrepo-0" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG [...] 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "758779904" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:16 INFO [http_cmd]: http status code OK 13:57:16 DEBUG --- status code --- 13:57:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:57:16 DEBUG --- http response --- 13:57:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:57:16 DEBUG ----- output ----- 13:57:16 DEBUG { 13:57:16 DEBUG "status": "success", 13:57:16 DEBUG "data": { 13:57:16 DEBUG "resultType": "vector", 13:57:16 DEBUG "result": [ 13:57:16 DEBUG { 13:57:16 DEBUG "metric": { 13:57:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG [...] 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG 13:57:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:57:16 DEBUG }, 13:57:16 DEBUG "value": [ 13:57:16 DEBUG 1735912636, 13:57:16 DEBUG "14987231232" 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG ] 13:57:16 DEBUG } 13:57:16 DEBUG } 13:57:28 INFO 13:57:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:57:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:57:28 INFO [loop_until]: OK (rc = 0) 13:57:28 DEBUG --- stdout --- 13:57:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 109m 0% 3841Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 82m 0% 19058Mi 67% 13:57:28 DEBUG --- stderr --- 13:57:28 DEBUG 13:57:31 INFO 13:57:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:57:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:57:31 INFO [loop_until]: OK (rc = 0) 13:57:31 DEBUG --- stdout --- 13:57:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:57:31 DEBUG --- stderr --- 13:57:31 DEBUG 13:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912695" 13:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912695" 13:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912695" 13:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912695" 13:58:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912695" 13:58:15 INFO [http_cmd]: http status code OK 13:58:15 DEBUG --- status code --- 13:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:15 DEBUG --- http response --- 13:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:15 ERROR [http_cmd]: ERROR 13:58:15 ERROR --- status code --- 13:58:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:58:15 INFO [http_cmd]: http status code OK 13:58:15 ERROR --- http response --- 13:58:15 DEBUG --- status code --- 13:58:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:58:15 DEBUG --- http response --- 13:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:15 INFO [http_cmd]: http status code OK 13:58:15 INFO [http_cmd]: http status code OK 13:58:15 DEBUG --- status code --- 13:58:15 DEBUG --- status code --- 13:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:15 DEBUG --- http response --- 13:58:15 DEBUG --- http response --- 13:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- http response --- 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 DEBUG --- status code --- 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "pod": "ds-cts-0" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "pod": "ds-idrepo-0" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "0.004351636255952868" 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "instance": "10.142.0.11:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "instance": "10.142.0.51:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "0.0003999999999981204" 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "pod": "ds-cts-0" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "pod": "ds-idrepo-0" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "0" 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "instance": "10.142.0.11:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "instance": "10.142.0.51:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "75532.5" 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "pod": "ds-cts-0" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "pod": "ds-idrepo-0" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "732.2361717980149" 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "instance": "10.142.0.11:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "instance": "10.142.0.51:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "92385.76666666668" 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:58:16 DEBUG "instance": "10.142.0.11:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:58:16 DEBUG "instance": "10.142.0.51:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "0.039832427641953175" 13:58:16 DEBUG "22118.399999999998" 13:58:16 DEBUG ] 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "instance": "10.142.0.11:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "instance": "10.142.0.51:9100" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "0.00033333333333350387" 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912696" 13:58:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912696" 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "pod": "ds-idrepo-0" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "758779904" 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:16 INFO [http_cmd]: http status code OK 13:58:16 DEBUG --- status code --- 13:58:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:58:16 DEBUG --- http response --- 13:58:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:58:16 DEBUG ----- output ----- 13:58:16 DEBUG { 13:58:16 DEBUG "status": "success", 13:58:16 DEBUG "data": { 13:58:16 DEBUG "resultType": "vector", 13:58:16 DEBUG "result": [ 13:58:16 DEBUG { 13:58:16 DEBUG "metric": { 13:58:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG [...] 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG 13:58:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:58:16 DEBUG }, 13:58:16 DEBUG "value": [ 13:58:16 DEBUG 1735912696, 13:58:16 DEBUG "14987501568" 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG ] 13:58:16 DEBUG } 13:58:16 DEBUG } 13:58:28 INFO 13:58:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:58:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:58:28 INFO [loop_until]: OK (rc = 0) 13:58:28 DEBUG --- stdout --- 13:58:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 113m 0% 3851Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 80m 0% 19059Mi 67% 13:58:28 DEBUG --- stderr --- 13:58:28 DEBUG 13:58:31 INFO 13:58:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:58:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:58:31 INFO [loop_until]: OK (rc = 0) 13:58:31 DEBUG --- stdout --- 13:58:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:58:31 DEBUG --- stderr --- 13:58:31 DEBUG 13:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912755" 13:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912755" 13:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912755" 13:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912755" 13:59:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912755" 13:59:15 INFO [http_cmd]: http status code OK 13:59:15 DEBUG --- status code --- 13:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:15 DEBUG --- http response --- 13:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:15 INFO [http_cmd]: http status code OK 13:59:15 DEBUG --- status code --- 13:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:15 DEBUG --- http response --- 13:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:15 INFO [http_cmd]: http status code OK 13:59:15 DEBUG --- status code --- 13:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:15 DEBUG --- http response --- 13:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:15 ERROR [http_cmd]: ERROR 13:59:15 ERROR --- status code --- 13:59:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:59:15 ERROR --- http response --- 13:59:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:59:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:59:15 INFO [http_cmd]: http status code OK 13:59:15 DEBUG --- status code --- 13:59:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:15 DEBUG --- http response --- 13:59:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 DEBUG --- http response --- 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 DEBUG --- status code --- 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:59:16 DEBUG ----- output ----- 13:59:16 DEBUG { 13:59:16 DEBUG "status": "success", 13:59:16 DEBUG "data": { 13:59:16 DEBUG "resultType": "vector", 13:59:16 DEBUG "result": [ 13:59:16 DEBUG { 13:59:16 DEBUG "metric": { 13:59:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG [...] 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG "pod": "ds-cts-0" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 1735912756, 13:59:16 DEBUG "0" 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:59:16 DEBUG ----- output ----- 13:59:16 DEBUG { 13:59:16 DEBUG "status": "success", 13:59:16 DEBUG "data": { 13:59:16 DEBUG "resultType": "vector", 13:59:16 DEBUG "result": [ 13:59:16 DEBUG { 13:59:16 DEBUG "metric": { 13:59:16 DEBUG "instance": "10.142.0.11:9100" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG [...] 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG "instance": "10.142.0.51:9100" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 1735912756, 13:59:16 DEBUG "0.0004000000000019099" 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-idrepo-0" }, "value": [ 1735912756, "0" ] } ] } } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "ds-idrepo-0" }, "value": [ 1735912756, "719.3757039090372" ] } ] } } 13:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:59:16 DEBUG ----- output ----- 13:59:16 DEBUG { 13:59:16 DEBUG "status": "success", 13:59:16 DEBUG "data": { 13:59:16 DEBUG "resultType": "vector", 13:59:16 DEBUG "result": [ 13:59:16 DEBUG { 13:59:16 DEBUG "metric": { 13:59:16 DEBUG "instance": "10.142.0.11:9100" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG [...] 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG "instance": "10.142.0.51:9100" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 1735912756, 13:59:16 DEBUG "80696.63333333335" 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:59:16 DEBUG ----- output ----- 13:59:16 DEBUG { 13:59:16 DEBUG "status": "success", 13:59:16 DEBUG "data": { 13:59:16 DEBUG "resultType": "vector", 13:59:16 DEBUG "result": [ 13:59:16 DEBUG { 13:59:16 DEBUG "metric": { 13:59:16 DEBUG "instance": "10.142.0.11:9100" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG [...] 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG "instance": "10.142.0.51:9100" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 1735912756, 13:59:16 DEBUG "98115.13333333333" 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912756" 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG ----- output ----- 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG { 13:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:59:16 DEBUG "status": "success", 13:59:16 DEBUG ----- output ----- 13:59:16 DEBUG "data": { 13:59:16 DEBUG { 13:59:16 DEBUG "resultType": "vector", 13:59:16 DEBUG "status": "success", 13:59:16 DEBUG "result": [ 13:59:16 DEBUG "data": { 13:59:16 DEBUG { 13:59:16 DEBUG "resultType": "vector", 13:59:16 DEBUG "metric": { 13:59:16 DEBUG "result": [ 13:59:16 DEBUG "instance": "10.142.0.11:9100" 13:59:16 DEBUG { 13:59:16 DEBUG }, 13:59:16 DEBUG "metric": { 13:59:16 DEBUG "value": [ 13:59:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:59:16 DEBUG 13:59:16 DEBUG }, 13:59:16 DEBUG 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG [...] 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG [...] 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG "instance": "10.142.0.51:9100" 13:59:16 DEBUG 13:59:16 DEBUG }, 13:59:16 DEBUG 13:59:16 DEBUG "value": [ 13:59:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:59:16 DEBUG 1735912756, 13:59:16 DEBUG }, 13:59:16 DEBUG "20343.466666666664" 13:59:16 DEBUG "value": [ 13:59:16 DEBUG ] 13:59:16 DEBUG 1735912756, 13:59:16 DEBUG } 13:59:16 DEBUG "0.04089239406421246" 13:59:16 DEBUG ] 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG } 13:59:16 DEBUG } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG ] 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG } 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG } 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:59:16 DEBUG ----- output ----- 13:59:16 DEBUG { 13:59:16 DEBUG "status": "success", 13:59:16 DEBUG "data": { 13:59:16 DEBUG "resultType": "vector", 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912756" 13:59:16 DEBUG "result": [ 13:59:16 DEBUG { 13:59:16 DEBUG "metric": { 13:59:16 DEBUG "instance": "10.142.0.11:9100" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG [...] 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG "instance": "10.142.0.51:9100" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 1735912756, 13:59:16 DEBUG "0.000666666666666534" 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG } 13:59:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912756" 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 13:59:16 DEBUG ----- output ----- 13:59:16 DEBUG { 13:59:16 DEBUG "status": "success", 13:59:16 DEBUG "data": { 13:59:16 DEBUG "resultType": "vector", 13:59:16 DEBUG "result": [ 13:59:16 DEBUG { 13:59:16 DEBUG "metric": { 13:59:16 DEBUG "pod": "ds-idrepo-0" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG [...] 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 1735912756, 13:59:16 DEBUG "758779904" 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG } 13:59:16 INFO [http_cmd]: http status code OK 13:59:16 DEBUG --- status code --- 13:59:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:59:16 DEBUG --- http response --- 13:59:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 13:59:16 DEBUG ----- output ----- 13:59:16 DEBUG { 13:59:16 DEBUG "status": "success", 13:59:16 DEBUG "data": { 13:59:16 DEBUG "resultType": "vector", 13:59:16 DEBUG "result": [ 13:59:16 DEBUG { 13:59:16 DEBUG "metric": { 13:59:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG [...] 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG 13:59:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 13:59:16 DEBUG }, 13:59:16 DEBUG "value": [ 13:59:16 DEBUG 1735912756, 13:59:16 DEBUG "14987669504" 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG ] 13:59:16 DEBUG } 13:59:16 DEBUG } 13:59:28 INFO 13:59:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 13:59:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:59:28 INFO [loop_until]: OK (rc = 0) 13:59:28 DEBUG --- stdout --- 13:59:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 101m 0% 3861Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 76m 0% 19063Mi 67% 13:59:28 DEBUG --- stderr --- 13:59:28 DEBUG 13:59:31 INFO 13:59:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 13:59:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:59:31 INFO [loop_until]: OK (rc = 0) 13:59:31 DEBUG --- stdout --- 13:59:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 537Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 13:59:31 DEBUG --- stderr --- 13:59:31 DEBUG 14:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912815" 14:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1735912815" 14:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912815" 14:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912815" 14:00:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912815" 14:00:15 INFO [http_cmd]: http status code OK 14:00:15 DEBUG --- status code --- 14:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:15 DEBUG --- http response --- 14:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:15 INFO [http_cmd]: http status code OK 14:00:15 DEBUG --- status code --- 14:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:15 DEBUG --- http response --- 14:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:15 INFO [http_cmd]: http status code OK 14:00:15 DEBUG --- status code --- 14:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:15 DEBUG --- http response --- 14:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:15 INFO [http_cmd]: http status code OK 14:00:15 DEBUG --- status code --- 14:00:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:15 DEBUG --- http response --- 14:00:15 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:15 ERROR [http_cmd]: ERROR 14:00:15 ERROR --- status code --- 14:00:15 ERROR http status code is 400 (expected [200, 401, 403, 404]) 14:00:15 ERROR --- http response --- 14:00:15 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 14:00:15 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG --- status code --- 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG --- status code --- 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG --- http response --- 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG { 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG "data": { 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG "result": [ 14:00:16 DEBUG { 14:00:16 DEBUG "metric": { 14:00:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG [...] 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG "pod": "ds-cts-0" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "0.0027444284325819934" 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG { 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG "data": { 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG "result": [ 14:00:16 DEBUG { 14:00:16 DEBUG "metric": { 14:00:16 DEBUG "instance": "10.142.0.11:9100" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG [...] 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG "instance": "10.142.0.51:9100" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "0.00023333333333539486" 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG } 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG { 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG "data": { 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1735912816" 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG "result": [ 14:00:16 DEBUG { 14:00:16 DEBUG "metric": { 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912816" 14:00:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG [...] 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG "pod": "ds-cts-0" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "0" 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG } 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG { 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG "data": { 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG "result": [ 14:00:16 DEBUG { 14:00:16 DEBUG "metric": { 14:00:16 DEBUG "pod": "rcs-645c86fdd6-j4rsk" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG [...] 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG "pod": "ds-cts-0" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "411.81004725359435" 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG } 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG { 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG "data": { 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG "result": [ 14:00:16 DEBUG { 14:00:16 DEBUG "metric": { 14:00:16 DEBUG "instance": "10.142.0.11:9100" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG [...] 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG "instance": "10.142.0.51:9100" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "74538.26666666668" 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG { 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG "data": { 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG "result": [ 14:00:16 DEBUG { 14:00:16 DEBUG "metric": { 14:00:16 DEBUG "instance": "10.142.0.11:9100" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG [...] 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG "instance": "10.142.0.51:9100" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "93997.29999999997" 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1735912816" 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG { 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG "data": { 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG "result": [ 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 14:00:16 DEBUG { 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG "metric": { 14:00:16 DEBUG { 14:00:16 DEBUG "instance": "10.142.0.11:9100" 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG }, 14:00:16 DEBUG "data": { 14:00:16 DEBUG "value": [ 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG 14:00:16 DEBUG "result": [ 14:00:16 DEBUG 14:00:16 DEBUG { 14:00:16 DEBUG 14:00:16 DEBUG "metric": { 14:00:16 DEBUG [...] 14:00:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1735912816" 14:00:16 DEBUG 14:00:16 DEBUG }, 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG 14:00:16 DEBUG "value": [ 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG "instance": "10.142.0.51:9100" 14:00:16 DEBUG 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 14:00:16 DEBUG }, 14:00:16 DEBUG 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG "value": [ 14:00:16 DEBUG [...] 14:00:16 DEBUG { 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG "4369.066666666667" 14:00:16 DEBUG 14:00:16 DEBUG "data": { 14:00:16 DEBUG ] 14:00:16 DEBUG 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG } 14:00:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 14:00:16 DEBUG "result": [ 14:00:16 DEBUG ] 14:00:16 DEBUG }, 14:00:16 DEBUG { 14:00:16 DEBUG } 14:00:16 DEBUG "value": [ 14:00:16 DEBUG "metric": { 14:00:16 DEBUG } 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "instance": "10.142.0.11:9100" 14:00:16 DEBUG "0.04026333213254849" 14:00:16 DEBUG }, 14:00:16 DEBUG ] 14:00:16 DEBUG "value": [ 14:00:16 DEBUG } 14:00:16 DEBUG 14:00:16 DEBUG ] 14:00:16 DEBUG 14:00:16 DEBUG } 14:00:16 DEBUG 14:00:16 DEBUG } 14:00:16 DEBUG [...] 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG "instance": "10.142.0.51:9100" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "0.0003333333333330302" 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG } 14:00:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1735912816" 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG { 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG "data": { 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG "result": [ 14:00:16 DEBUG { 14:00:16 DEBUG "metric": { 14:00:16 DEBUG "pod": "ds-idrepo-0" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG [...] 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG "pod": "rcs-645c86fdd6-m29zn" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "758779904" 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG } 14:00:16 INFO [http_cmd]: http status code OK 14:00:16 DEBUG --- status code --- 14:00:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 14:00:16 DEBUG --- http response --- 14:00:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 14:00:16 DEBUG ----- output ----- 14:00:16 DEBUG { 14:00:16 DEBUG "status": "success", 14:00:16 DEBUG "data": { 14:00:16 DEBUG "resultType": "vector", 14:00:16 DEBUG "result": [ 14:00:16 DEBUG { 14:00:16 DEBUG "metric": { 14:00:16 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG [...] 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG 14:00:16 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 14:00:16 DEBUG }, 14:00:16 DEBUG "value": [ 14:00:16 DEBUG 1735912816, 14:00:16 DEBUG "14986067968" 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG ] 14:00:16 DEBUG } 14:00:16 DEBUG } 14:00:28 INFO 14:00:28 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 14:00:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 14:00:28 INFO [loop_until]: OK (rc = 0) 14:00:28 DEBUG --- stdout --- 14:00:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 105m 0% 3856Mi 6% gke-xlou-cdm-frontend-18a44ad7-wxs9 73m 0% 19059Mi 67% 14:00:28 DEBUG --- stderr --- 14:00:28 DEBUG 14:00:31 INFO 14:00:31 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 14:00:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 14:00:31 INFO [loop_until]: OK (rc = 0) 14:00:31 DEBUG --- stdout --- 14:00:31 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 3m 536Mi ds-idrepo-0 7m 13477Mi rcs-645c86fdd6-j4rsk 1m 822Mi rcs-645c86fdd6-m29zn 1m 361Mi 14:00:31 DEBUG --- stderr --- 14:00:31 DEBUG 14:00:36 WARNING File /tmp/lodemon_data-20241228_071044/node_disk_read_bytes_total_file.json does not exist. Skipping average value computation... 127.0.0.1 - - [03/Jan/2025 14:00:36] "GET /monitoring/average?start_time=20241228_071357&stop_time=20250103_135835 HTTP/1.1" 200 - 14:00:48 INFO Finished: True 14:00:48 INFO Waiting for threads to register finish flag 14:01:31 INFO Done. Have a nice day! :) 127.0.0.1 - - [03/Jan/2025 14:01:31] "GET /monitoring/stop HTTP/1.1" 200 - 14:01:37 WARNING File /tmp/lodemon_data-20241228_071044/node_disk_read_bytes_total_file.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3087', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3038', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2597', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2891', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3254', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2793', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3401', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2205', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2156', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3450', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2450', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3352', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2352', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2548', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3136', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2695', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2401', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2499', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2646', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2842', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2940', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2744', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2303', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2254', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3303', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2989', ...) 127.0.0.1 - - [03/Jan/2025 14:01:41] "GET /monitoring/process HTTP/1.1" 200 -